[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 436251520 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001019] APIC: Switch to symmetric I/O mode setup [ 0.003390] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005021] kvm-guest: setup PV IPIs [ 0.008592] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011139] LSM: Security Framework initializing [ 0.012061] Yama: becoming mindful. [ 0.013043] SELinux: Initializing. [ 0.014101] *** VALIDATE selinux *** [ 0.023134] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028130] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031117] *** VALIDATE tmpfs *** [ 0.032492] *** VALIDATE proc *** [ 0.034122] *** VALIDATE cgroup *** [ 0.035012] *** VALIDATE cgroup2 *** [ 0.037288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040029] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.044122] debug: unmapping init [mem 0xffffffffa7e59000-0xffffffffa7e60fff] [ 0.046173] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047685] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048027] ... version: 2 [ 0.049010] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.050989] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054297] rcu: Hierarchical SRCU implementation. [ 0.056536] smp: Bringing up secondary CPUs ... [ 0.057649] x86: Booting SMP configuration: [ 0.058026] .... node #0, CPUs: #1 #2 #3 [ 0.061362] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.092019] node 0 deferred pages initialised in 25ms [ 0.095235] devtmpfs: initialized [ 0.096280] x86/mm: Memory block size: 128MB [ 0.098713] gcov: version magic: 0x41383552 [ 0.100211] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.101444] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.102235] pinctrl core: initialized pinctrl subsystem [ 0.103191] [ 0.103667] ************************************************************* [ 0.104014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.105010] ** ** [ 0.106013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.107013] ** ** [ 0.108017] ** This means that this kernel is built to expose internal ** [ 0.109013] ** IOMMU data structures, which may compromise security on ** [ 0.110010] ** your system. ** [ 0.111010] ** ** [ 0.112009] ** If you see this message and you are not debugging the ** [ 0.113010] ** kernel, report this immediately to your vendor! ** [ 0.114010] ** ** [ 0.115009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.116009] ************************************************************* [ 0.117728] NET: Registered protocol family 16 [ 0.118438] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.119050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.120064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.121416] cpuidle: using governor menu [ 0.123792] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.126465] PCI: Using configuration type 1 for base access [ 0.128129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.138115] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.141108] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.145087] cryptd: max_cpu_qlen set to 1000 [ 0.148265] ACPI: Added _OSI(Module Device) [ 0.150015] ACPI: Added _OSI(Processor Device) [ 0.151009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.152011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.157729] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.163537] ACPI: Interpreter enabled [ 0.165113] ACPI: PM: (supports S0 S3 S4 S5) [ 0.169032] ACPI: Using IOAPIC for interrupt routing [ 0.171795] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.176424] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.187728] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.191047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.194027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.200097] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.205302] acpiphp: Slot [2] registered [ 0.207125] acpiphp: Slot [5] registered [ 0.209132] acpiphp: Slot [6] registered [ 0.210105] acpiphp: Slot [3] registered [ 0.212132] acpiphp: Slot [4] registered [ 0.215116] acpiphp: Slot [7] registered [ 0.217111] acpiphp: Slot [8] registered [ 0.218092] acpiphp: Slot [9] registered [ 0.220119] acpiphp: Slot [10] registered [ 0.221112] acpiphp: Slot [11] registered [ 0.223098] acpiphp: Slot [12] registered [ 0.224098] acpiphp: Slot [13] registered [ 0.226101] acpiphp: Slot [14] registered [ 0.228107] acpiphp: Slot [15] registered [ 0.229101] acpiphp: Slot [16] registered [ 0.231132] acpiphp: Slot [17] registered [ 0.232139] acpiphp: Slot [18] registered [ 0.235299] acpiphp: Slot [19] registered [ 0.237153] acpiphp: Slot [20] registered [ 0.239152] acpiphp: Slot [21] registered [ 0.241125] acpiphp: Slot [22] registered [ 0.243125] acpiphp: Slot [23] registered [ 0.244069] acpiphp: Slot [24] registered [ 0.245121] acpiphp: Slot [25] registered [ 0.247307] acpiphp: Slot [26] registered [ 0.249115] acpiphp: Slot [27] registered [ 0.251167] acpiphp: Slot [28] registered [ 0.253149] acpiphp: Slot [29] registered [ 0.254102] acpiphp: Slot [30] registered [ 0.256131] acpiphp: Slot [31] registered [ 0.258062] PCI host bridge to bus 0000:00 [ 0.260022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.263030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.266015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.268025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.272028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.276027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.279584] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.283291] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.287346] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.295604] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.302050] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.305016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.307019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.310018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.315145] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.321705] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.326066] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.332179] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.337022] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.349024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.355022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.361733] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.368022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.373019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.385967] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.395891] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.402083] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.409027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.424014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.434374] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.436466] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.439419] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.441477] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.443293] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.448147] iommu: Default domain type: Passthrough [ 0.450485] SCSI subsystem initialized [ 0.451155] ACPI: bus type USB registered [ 0.453110] usbcore: registered new interface driver usbfs [ 0.455126] usbcore: registered new interface driver hub [ 0.457094] usbcore: registered new device driver usb [ 0.459174] pps_core: LinuxPPS API ver. 1 registered [ 0.461014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.465072] PTP clock support registered [ 0.467271] EDAC MC: Ver: 3.0.0 [ 0.469175] PCI: Using ACPI for IRQ routing [ 0.470909] NetLabel: Initializing [ 0.472032] NetLabel: domain hash size = 128 [ 0.474018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.477094] NetLabel: unlabeled traffic allowed by default [ 0.480183] vgaarb: loaded [ 0.482376] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.484023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.490000] clocksource: Switched to clocksource kvm-clock [ 0.606122] VFS: Disk quotas dquot_6.6.0 [ 0.608436] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.611561] *** VALIDATE ramfs *** [ 0.613114] *** VALIDATE hugetlbfs *** [ 0.615191] pnp: PnP ACPI init [ 0.618995] pnp: PnP ACPI: found 6 devices [ 0.638844] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.642850] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.645515] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.648135] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.651088] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.653794] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.657069] NET: Registered protocol family 2 [ 0.659970] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.666116] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.669879] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.675873] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.679646] TCP: Hash tables configured (established 65536 bind 65536) [ 0.683237] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.687542] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.690683] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.694144] NET: Registered protocol family 1 [ 0.697119] RPC: Registered named UNIX socket transport module. [ 0.699456] RPC: Registered udp transport module. [ 0.700873] RPC: Registered tcp transport module. [ 0.702442] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.704965] NET: Registered protocol family 44 [ 0.706957] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.709598] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.712152] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.714754] PCI: CLS 0 bytes, default 64 [ 0.716736] Unpacking initramfs... [ 2.185720] debug: unmapping init [mem 0xffff9606bcc64000-0xffff9606bffcffff] [ 2.193240] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.196182] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.201242] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.708614] Initialise system trusted keyrings [ 2.711723] Key type blacklist registered [ 2.713757] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.723275] zbud: loaded [ 2.727485] *** VALIDATE nfs *** [ 2.729073] *** VALIDATE nfs4 *** [ 2.731140] pstore: using deflate compression [ 2.734976] Platform Keyring initialized [ 2.840453] NET: Registered protocol family 38 [ 2.842641] Key type asymmetric registered [ 2.844042] Asymmetric key parser 'x509' registered [ 2.846278] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.849637] io scheduler mq-deadline registered [ 2.851738] io scheduler kyber registered [ 2.853701] io scheduler bfq registered [ 2.856252] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.859990] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.863490] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.867266] ACPI: Power Button [PWRF] [ 2.873922] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.881766] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.894339] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.923134] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.955200] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.973731] Non-volatile memory driver v1.3 [ 2.975899] Linux agpgart interface v0.103 [ 3.016406] virtio_blk virtio1: [vda] 133936 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.019907] vda: detected capacity change from 0 to 68575232 [ 3.040791] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.043637] vdb: detected capacity change from 0 to 1073741824 [ 3.052388] libphy: Fixed MDIO Bus: probed [ 3.060262] usbcore: registered new interface driver usbserial_generic [ 3.063708] usbserial: USB Serial support registered for generic [ 3.066668] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.072020] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.074051] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.076771] mousedev: PS/2 mouse device common for all mice [ 3.080176] rtc_cmos 00:05: RTC can wake from S4 [ 3.083500] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.084457] rtc_cmos 00:05: registered as rtc0 [ 3.093349] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.098443] intel_pstate: CPU model not supported [ 3.101751] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.105937] hid: raw HID events driver (C) Jiri Kosina [ 3.109123] usbcore: registered new interface driver usbhid [ 3.109222] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.111878] usbhid: USB HID core driver [ 3.117256] drop_monitor: Initializing network drop monitor service [ 3.119722] Initializing XFRM netlink socket [ 3.121740] NET: Registered protocol family 10 [ 3.124468] Segment Routing with IPv6 [ 3.126050] NET: Registered protocol family 17 [ 3.128444] mpls_gso: MPLS GSO support [ 3.134406] RAS: Correctable Errors collector initialized. [ 3.136792] AVX version of gcm_enc/dec engaged. [ 3.138628] AES CTR mode by8 optimization enabled [ 3.225052] sched_clock: Marking stable (3225023447, 0)->(4174804872, -949781425) [ 3.228786] registered taskstats version 1 [ 3.231290] Loading compiled-in X.509 certificates [ 3.233843] zswap: loaded using pool lzo/zbud [ 3.260575] Key type big_key registered [ 3.272767] Key type encrypted registered [ 3.274690] ima: No TPM chip found, activating TPM-bypass! [ 3.276893] ima: Allocated hash algorithm: sha1 [ 3.278601] ima: No architecture policies found [ 3.279935] evm: Initialising EVM extended attributes: [ 3.282233] evm: security.selinux [ 3.283640] evm: security.ima [ 3.284453] evm: security.capability [ 3.286247] evm: HMAC attrs: 0x1 [ 3.288988] rtc_cmos 00:05: setting system clock to 2026-04-09 19:36:27 UTC (1775763387) [ 3.298364] debug: unmapping init [mem 0xffffffffa8e03000-0xffffffffa8ffffff] [ 3.301405] debug: unmapping init [mem 0xffffffffa7b82000-0xffffffffa7e58fff] [ 3.308166] Write protecting the kernel read-only data: 28672k [ 3.312261] debug: unmapping init [mem 0xffffffffa6203000-0xffffffffa63fffff] [ 3.315636] debug: unmapping init [mem 0xffffffffa6b14000-0xffffffffa6bfffff] [ 3.351791] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.362133] systemd[1]: Detected virtualization kvm. [ 3.364386] systemd[1]: Detected architecture x86-64. [ 3.366689] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.389694] systemd[1]: No hostname configured. [ 3.391782] systemd[1]: Set hostname to . [ 3.394423] random: systemd: uninitialized urandom read (16 bytes read) [ 3.396196] systemd[1]: Initializing machine ID from random generator. [ 3.444771] random: ln: uninitialized urandom read (6 bytes read) [ 3.549729] random: systemd: uninitialized urandom read (16 bytes read) [ 3.553201] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.557609] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.563064] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.185664] device-mapper: uevent: version 1.0.3 [ 4.188257] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.013956] virtio_net virtio0 ens2: renamed from eth0 [ 5.047752] random: fast init done [ 5.080694] scsi host0: ata_piix [ 5.157281] scsi host1: ata_piix [ 5.181910] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.185136] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.959244] dracut-initqueue[598]: RTNETLINK answers: File exists [ 9.793273] random: crng init done [ 9.794660] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.306745] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.558341] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.813306] SELinux: Disabled at runtime. [ 11.874329] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.883642] systemd[1]: Detected virtualization kvm. [ 11.885718] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.382591] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.386774] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.391889] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.399314] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.402735] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.410209] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.428066] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.467831] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.812659] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.226550] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.249958] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.420140] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.490327] EDAC sbridge: Ver: 1.1.2 [ 14.859395] Key type dns_resolver registered [ 15.159149] NFS: Registering the id_resolver key type [ 15.161547] Key type id_resolver registered [ 15.163498] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg445-client login: [ 43.898266] libcfs: loading out-of-tree module taints kernel. [ 43.960229] Key type ._llcrypt registered [ 43.961876] Key type .llcrypt registered [ 44.199292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 44.206193] alg: No test for adler32 (adler32-zlib) [ 45.236146] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 45.588946] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 47.223209] Key type lgssc registered [ 47.890994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 172.225507] Lustre: Mounted lustre-client [ 177.592281] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 198.111830] Lustre: lustre-OST0000-osc-ffff960709f43000: disconnect after 24s idle [ 198.948363] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing check_logdir /tmp/testlogs/ [ 204.934496] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing yml_node [ 209.692248] Lustre: DEBUG MARKER: Client: 2.17.51.76 [ 212.247701] Lustre: DEBUG MARKER: MDS: 2.17.51.76 [ 215.099826] Lustre: DEBUG MARKER: OSS: 2.17.51.76 [ 216.694963] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Apr 9 15:39:59 EDT 2026 [ 233.722726] LustreError: 7937:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff960709f43000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 233.770359] Lustre: Unmounted lustre-client [ 255.738401] Key type lgssc unregistered [ 256.031833] LNet: 8384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 256.039159] LNetError: 8384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 256.071130] LNet: Removed LNI 192.168.204.45@tcp [ 256.849157] Key type .llcrypt unregistered [ 256.851055] Key type ._llcrypt unregistered [ 284.850731] Key type ._llcrypt registered [ 284.856281] Key type .llcrypt registered [ 285.087217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 285.119642] alg: No test for adler32 (adler32-zlib) [ 286.079481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 286.140303] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 286.169867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 290.917943] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 291.027520] hrtimer: interrupt took 6510021 ns [ 291.157164] LNet: 9262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 291.168489] LNetError: 9262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 291.185954] LNet: Removed LNI 192.168.204.45@tcp [ 292.016404] Key type .llcrypt unregistered [ 292.020120] Key type ._llcrypt unregistered [ 298.454387] Key type ._llcrypt registered [ 298.457498] Key type .llcrypt registered [ 299.574817] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 15:41:22 (1775763682) [ 299.933837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 299.955898] alg: No test for adler32 (adler32-zlib) [ 300.858969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.945600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.963806] LNet: 9674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.982879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.040642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 301.056198] LNet: 9681:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 306.497354] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 15:41:29 (1775763689) [ 306.737346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.766713] LNet: 10258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.798658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.872499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 313.706981] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 15:41:36 (1775763696) [ 313.822115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.865218] LNet: 10837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.897857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.008359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 321.570217] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 15:41:44 (1775763704) [ 321.915775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 321.953400] LNet: 11428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.992471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.122797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 330.482943] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 15:41:52 (1775763712) [ 330.710493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.768477] LNet: 12009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.822173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.016919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 331.815272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 341.486351] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 15:42:03 (1775763723) [ 341.640728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.687276] LNet: 12600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.735693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.821756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 342.658985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 343.080762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 350.696692] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 15:42:13 (1775763733) [ 350.813158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.897172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.012922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 358.281228] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 15:42:21 (1775763741) [ 358.587408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.631330] LNet: 13789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.639289] LNet: 13789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 358.681594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.801866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 358.849743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 359.287823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 359.360102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 359.899274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 359.937784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 360.384547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 360.429475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 361.046903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 361.106911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 361.744498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 361.791309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 362.389138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 362.438509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 363.129548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 363.181593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 363.850481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 363.905687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 364.491940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 364.582294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 365.187877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 365.278487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 365.808205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 365.872541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 366.396379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 366.447587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 373.272753] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 15:42:36 (1775763756) [ 373.405861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.470139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.568751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 373.610850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 379.876600] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 15:42:42 (1775763762) [ 380.012385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.071643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.169568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 380.205212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 386.599736] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 15:42:49 (1775763769) [ 386.992397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.052405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.154527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 387.235904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 393.776527] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 15:42:56 (1775763776) [ 394.107539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.142367] LNet: 16231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.149800] LNet: 16231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 394.176725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.271031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 394.335044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 400.969498] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 15:43:03 (1775763783) [ 401.207387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.258701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.404471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 401.450586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 408.565851] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 15:43:11 (1775763791) [ 408.791369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.835481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.931590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 408.980783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 415.126386] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 15:43:18 (1775763798) [ 415.473797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.510181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.578796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 415.970407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 422.113304] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 15:43:25 (1775763805) [ 422.425113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.490875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.786788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 423.022488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 429.960332] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 15:43:32 (1775763812) [ 430.066269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.128271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.363432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 430.704146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 438.493889] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 15:43:41 (1775763821) [ 438.734208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.796427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.346986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 439.682797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 446.632761] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 15:43:49 (1775763829) [ 446.904617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.945138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.018517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 447.080595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 454.787188] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 15:43:57 (1775763837) [ 455.305711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.393349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.502618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 455.578599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 463.759777] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 15:44:06 (1775763846) [ 463.861379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.900258] LNet: 21666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 463.908191] LNet: 21666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 463.947913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.032541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 471.654352] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 15:44:14 (1775763854) [ 472.007872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.075557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.133971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 479.450964] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 15:44:22 (1775763862) [ 479.808276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.858688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.935581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 480.428036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 487.512087] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 15:44:30 (1775763870) [ 487.638262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.732181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.843906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 487.897611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 494.833384] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 15:44:37 (1775763877) [ 494.922477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.000092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.118484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 495.171880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 502.020914] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 15:44:44 (1775763884) [ 502.250128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.289364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.364679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 502.404626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 508.873653] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 15:44:51 (1775763891) [ 509.043351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.085596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.208664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 509.281977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 515.978947] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 15:44:58 (1775763898) [ 516.057736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.121854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.187921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 516.250678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 516.275231] LNetError: 25790:0:(peer.c:2174:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 516.318424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 516.375365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 521.229978] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 15:45:04 (1775763904) [ 521.323484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.367971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.496956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Dxil0A7Dli [ 521.581204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Dxil0A7Dli [ 521.648831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Dxil0A7Dli [ 528.343620] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 15:45:11 (1775763911) [ 528.902827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.971246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.015720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 529.044244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 529.082818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 536.230974] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 15:45:18 (1775763918) [ 536.337464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.448168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.537732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 539.613926] LNet: Message forwarding will be enabled in 2s [ 542.687278] LNet: Message forwarding enabled [ 543.155130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 543.192852] LNet: Message forwarding will stop in 2s to 6s [ 547.756938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 548.391713] LNet: Message forwarding disabled [ 550.244533] LNet: Message forwarding will be enabled in 2s [ 553.503131] LNet: Message forwarding enabled [ 556.983966] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 15:45:39 (1775763939) [ 557.172220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.778305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.881548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 565.616084] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 15:45:48 (1775763948) [ 565.900108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.937690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.012723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 566.044600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 566.080321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 566.119388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 566.154178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 566.186156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 566.214478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 566.246803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 566.276966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 566.311313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 566.353379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 566.394898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 566.438735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 566.484838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 566.549991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 566.604871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 566.649357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 566.693410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 566.723317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 566.757488] LNetError: 28861:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 572.999903] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 15:45:55 (1775763955) [ 573.330553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.391491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.472820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 579.627822] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 15:46:02 (1775763962) [ 581.552862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg445-server [ 583.203640] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 15:46:06 (1775763966) [ 583.478362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.524967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.593763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 583.623920] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 585.260785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[46-50]@tcp [ 585.285573] LNet: 30577:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 585.371330] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.46@tcp is being used as a gateway but routing feature is not turned on [ 585.391071] LNetError: 30367:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.46@tcp has gone from up to down [ 586.338468] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.46@tcp is being used as a gateway but routing feature is not turned on [ 586.368630] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 587.366026] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.46@tcp is being used as a gateway but routing feature is not turned on [ 587.377149] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 589.422552] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.48@tcp is being used as a gateway but routing feature is not turned on [ 589.446182] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 592.189977] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 15:46:15 (1775763975) [ 593.522354] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.47@tcp is being used as a gateway but routing feature is not turned on [ 593.532500] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 594.433818] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg445-server [ 596.352661] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 15:46:19 (1775763979) [ 596.509960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.568645] LNet: 31506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.580478] LNet: 31506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 596.589767] LNetError: 31506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.621787] LNet: Removed LNI 192.168.204.45@tcp [ 596.677338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.750431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 596.781476] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 597.351246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[46-51/2]@tcp [ 597.403093] LNet: 31629:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 597.421853] LNet: 31629:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 597.451488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[46-51/2]@tcp [ 597.457303] LNetError: 31516:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.48@tcp has gone from up to down [ 597.482711] LNetError: 31516:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 603.820442] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 15:46:26 (1775763986) [ 604.145310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.170150] LNetError: 32222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 604.199690] LNet: Removed LNI 192.168.204.45@tcp [ 604.244358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.317990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 604.355580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.403796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.473719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 604.541420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.568574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.635600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 604.705662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.740328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.818395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 604.933313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.979722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.057857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 605.154153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.182128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.264272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 605.356119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.399916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.491673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 605.582055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.619813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.671795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 605.750524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.817283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.885576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 605.939793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.978394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.054743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 606.152969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.204461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.289736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 606.359383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.390772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.438790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 611.948614] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 15:46:34 (1775763994) [ 613.651574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg445-server [ 615.221750] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 15:46:38 (1775763998) [ 617.045490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg445-server [ 618.589780] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 15:46:41 (1775764001) [ 619.456169] Key type .llcrypt unregistered [ 619.457954] Key type ._llcrypt unregistered [ 624.430059] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 624.592134] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 624.594661] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 624.597247] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 627.084155] Key type ._llcrypt registered [ 627.086227] Key type .llcrypt registered [ 627.285822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 627.300670] alg: No test for adler32 (adler32-zlib) [ 628.145108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.172761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 628.194648] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 629.032357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 629.077035] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 631.552475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 631.587583] LNet: Removed LNI 10.1.2.3@tcp [ 631.798218] LNet: 34340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 631.818954] LNetError: 34340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 631.847935] LNet: Removed LNI 192.168.204.45@tcp [ 632.775582] Key type .llcrypt unregistered [ 632.777213] Key type ._llcrypt unregistered [ 647.293842] Key type ._llcrypt registered [ 647.302258] Key type .llcrypt registered [ 648.688870] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 15:47:11 (1775764031) [ 649.548874] Key type .llcrypt unregistered [ 649.556298] Key type ._llcrypt unregistered [ 656.032095] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 656.357807] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 656.366513] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 656.380816] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 659.162423] Key type ._llcrypt registered [ 659.171412] Key type .llcrypt registered [ 659.409020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 659.424689] alg: No test for adler32 (adler32-zlib) [ 660.355207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.406518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 660.428805] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 661.013364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 661.026522] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 663.271352] LNet: 35834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.279552] LNetError: 35834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.291242] LNet: Removed LNI 192.168.204.45@tcp [ 663.302321] LNet: Removed LNI 10.1.2.3@tcp [ 663.846597] Key type .llcrypt unregistered [ 663.848355] Key type ._llcrypt unregistered [ 674.025572] Key type ._llcrypt registered [ 674.030476] Key type .llcrypt registered [ 675.456273] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 15:47:38 (1775764058) [ 675.893290] Key type .llcrypt unregistered [ 675.901250] Key type ._llcrypt unregistered [ 681.460816] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 681.650966] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 681.655934] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 681.659290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 684.251851] Key type ._llcrypt registered [ 684.254920] Key type .llcrypt registered [ 684.437663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 684.459707] alg: No test for adler32 (adler32-zlib) [ 685.383990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.466411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 685.486273] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 687.441040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 687.465896] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 689.384051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 689.404763] LNet: Removed LNI 10.1.2.3@tcp [ 689.418134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 689.445798] LNet: Removed LNI 10.1.2.31@tcp [ 689.459734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 689.479941] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 690.127681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 690.143893] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 691.079648] LNet: 37754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.091623] LNetError: 37754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 691.106817] LNet: Removed LNI 10.1.2.3@tcp [ 691.121710] LNet: Removed LNI 10.1.2.31@tcp [ 691.760741] Key type .llcrypt unregistered [ 691.764049] Key type ._llcrypt unregistered [ 704.066154] Key type ._llcrypt registered [ 704.068770] Key type .llcrypt registered [ 705.290048] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 15:48:08 (1775764088) [ 705.677750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 705.703504] alg: No test for adler32 (adler32-zlib) [ 706.696400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.767984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 706.807967] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 707.445528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.479911] LNet: 38708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.488629] LNetError: 38708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.503683] LNet: Removed LNI 192.168.204.45@tcp [ 707.518302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.555774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.45@tcp [ 707.577544] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 708.306306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.45@tcp [ 708.343355] LNet: Removed LNI 192.168.204.45@tcp [ 708.443888] LNet: 38843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.005125] Key type .llcrypt unregistered [ 709.009287] Key type ._llcrypt unregistered [ 721.070814] Key type ._llcrypt registered [ 721.073890] Key type .llcrypt registered [ 722.460792] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 15:48:25 (1775764105) [ 724.510724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg445-server [ 726.403884] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 15:48:29 (1775764109) [ 727.021988] Key type .llcrypt unregistered [ 727.033553] Key type ._llcrypt unregistered [ 732.980260] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 733.230794] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 733.246492] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 733.256189] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 735.874801] Key type ._llcrypt registered [ 735.879643] Key type .llcrypt registered [ 736.175812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 736.207190] alg: No test for adler32 (adler32-zlib) [ 737.151956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.286508] LNet: Added LNI 192.168.204.45@tcp1 [8/256/0/180] [ 738.248294] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 740.390257] LNet: 40681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.397587] LNetError: 40681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 740.407836] LNet: Removed LNI 192.168.204.45@tcp1 [ 740.418947] LNet: Removed LNI 10.1.2.3@tcp2 [ 741.149667] Key type .llcrypt unregistered [ 741.157866] Key type ._llcrypt unregistered [ 756.132041] Key type ._llcrypt registered [ 756.138471] Key type .llcrypt registered [ 758.097192] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 15:49:00 (1775764140) [ 758.574147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 758.587949] alg: No test for adler32 (adler32-zlib) [ 759.609712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.706395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 759.742815] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 760.960371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.995940] LNet: 41651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.003578] LNetError: 41651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 761.022827] LNet: Removed LNI 192.168.204.45@tcp [ 761.045633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.117721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 761.144805] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 767.945958] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 15:49:10 (1775764150) [ 768.098052] LNet: 42235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.112420] LNetError: 42235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.140709] LNet: Removed LNI 192.168.204.45@tcp [ 768.788537] Key type .llcrypt unregistered [ 768.794865] Key type ._llcrypt unregistered [ 774.519554] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 774.577702] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 774.584465] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 774.590098] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 777.421835] Key type ._llcrypt registered [ 777.429949] Key type .llcrypt registered [ 777.596893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 777.611595] alg: No test for adler32 (adler32-zlib) [ 778.454699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.489756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.505758] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 779.137827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 779.156505] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 781.637761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.670804] LNet: 42989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.682734] LNetError: 42989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.695685] LNet: Removed LNI 192.168.204.45@tcp [ 781.724398] LNet: Removed LNI 10.1.2.3@tcp [ 781.751792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.863929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 781.913454] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 781.926423] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 782.551589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 782.585125] LNet: 43016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.594862] LNetError: 43016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.617903] LNetError: 43016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 782.624148] LNet: Removed LNI 192.168.204.45@tcp [ 782.647884] LNet: Removed LNI 10.1.2.3@tcp [ 782.683399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.752073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 782.787214] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 782.798599] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 783.450083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 783.485156] LNet: Added LNI 192.168.204.45@tcp2 [8/256/0/180] [ 784.625085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.654614] LNet: 43179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.665401] LNetError: 43179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.682859] LNetError: 43179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 784.691018] LNet: Removed LNI 192.168.204.45@tcp [ 784.693157] LNet: Removed LNI 10.1.2.3@tcp [ 784.718337] LNet: Removed LNI 192.168.204.45@tcp2 [ 784.737306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.815737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 784.854407] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 784.862051] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 784.868625] LNet: Added LNI 192.168.204.45@tcp2 [8/256/0/180] [ 792.817371] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 15:49:35 (1775764175) [ 793.038476] LNet: 43776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.051849] LNetError: 43776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.064905] LNetError: 43776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 793.072738] LNet: Removed LNI 192.168.204.45@tcp [ 793.079389] LNet: Removed LNI 10.1.2.3@tcp [ 793.095403] LNet: Removed LNI 192.168.204.45@tcp2 [ 793.817065] Key type .llcrypt unregistered [ 793.826319] Key type ._llcrypt unregistered [ 800.204505] Key type ._llcrypt registered [ 800.208572] Key type .llcrypt registered [ 800.464752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 800.497271] alg: No test for adler32 (adler32-zlib) [ 801.418933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.484555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 801.507906] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 802.655816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.702952] LNet: 44172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.709858] LNetError: 44172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.747566] LNet: Removed LNI 192.168.204.45@tcp [ 802.784572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.882338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 802.903924] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 803.434138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.454255] LNet: 44202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.461976] LNetError: 44202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.482608] LNet: Removed LNI 192.168.204.45@tcp [ 803.516741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.595778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 803.624251] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 804.349260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.380423] LNet: Removed LNI 192.168.204.45@tcp [ 804.409975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.491740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 804.529329] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 805.086266] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 805.203042] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 805.207254] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 805.212113] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 807.718241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 807.738555] LNet: 44309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.744264] LNet: 44309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 807.753425] LNetError: 44309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.763916] LNetError: 44309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 807.778913] LNet: Removed LNI 192.168.204.45@tcp [ 807.801333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.859736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 807.891359] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 814.882416] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 15:49:57 (1775764197) [ 815.225528] LNet: 44907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.236632] LNetError: 44907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.255060] LNet: Removed LNI 192.168.204.45@tcp [ 815.936666] Key type .llcrypt unregistered [ 815.940738] Key type ._llcrypt unregistered [ 821.650394] Key type ._llcrypt registered [ 821.652335] Key type .llcrypt registered [ 821.880547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 821.909626] alg: No test for adler32 (adler32-zlib) [ 822.844271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.934216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 822.955312] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 823.742538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 823.760957] LNet: 45305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.766718] LNetError: 45305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.778671] LNet: Removed LNI 192.168.204.45@tcp [ 823.798227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.862437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 823.891378] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 830.202039] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 15:50:13 (1775764213) [ 830.505881] LNet: 45897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.523879] LNetError: 45897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.549593] LNet: Removed LNI 192.168.204.45@tcp [ 831.188477] Key type .llcrypt unregistered [ 831.190410] Key type ._llcrypt unregistered [ 836.789954] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 836.946752] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 836.963876] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 836.974956] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 839.981660] Key type ._llcrypt registered [ 839.986080] Key type .llcrypt registered [ 840.418103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 840.450041] alg: No test for adler32 (adler32-zlib) [ 841.337261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.407178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 841.448573] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 842.198686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.215621] LNet: 46343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.220482] LNetError: 46343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.244817] LNet: Removed LNI 192.168.204.45@tcp [ 842.261187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.314053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 842.335577] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 849.098391] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 15:50:31 (1775764231) [ 849.221874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 849.247200] LNet: 46936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.255605] LNetError: 46936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.279418] LNet: Removed LNI 192.168.204.45@tcp [ 849.318842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.523855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 855.686367] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 15:50:38 (1775764238) [ 856.104796] LNet: 47524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.752284] Key type .llcrypt unregistered [ 856.754443] Key type ._llcrypt unregistered [ 863.987657] Key type ._llcrypt registered [ 864.004829] Key type .llcrypt registered [ 864.405417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 864.425243] alg: No test for adler32 (adler32-zlib) [ 865.409709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.516295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 865.574733] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 866.757740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.784209] LNet: 47921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.790696] LNetError: 47921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.809193] LNet: Removed LNI 192.168.204.45@tcp [ 866.841730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.924460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 867.003741] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 874.162435] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 15:50:57 (1775764257) [ 874.444453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.479421] LNet: 48514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.488098] LNetError: 48514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.505969] LNet: Removed LNI 192.168.204.45@tcp [ 874.547582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.647560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 882.142861] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 15:51:04 (1775764264) [ 883.739957] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 885.847634] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 15:51:08 (1775764268) [ 887.686725] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 890.349607] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 15:51:12 (1775764272) [ 892.065032] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 893.748587] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 15:51:16 (1775764276) [ 895.243374] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 896.992858] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 15:51:19 (1775764279) [ 898.917863] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 900.774583] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 15:51:23 (1775764283) [ 900.890609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.925987] LNet: 50561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.002660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.144454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 901.183514] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 902.053693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.45 [ 902.085523] LNet: 50703:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 902.099767] LNet: Removed LNI 192.168.204.45@tcp [ 908.396473] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 15:51:31 (1775764291) [ 908.552778] LNet: 51265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.567292] LNetError: 51265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.592659] LNet: Removed LNI 192.168.204.45@tcp [ 909.253469] Key type .llcrypt unregistered [ 909.257168] Key type ._llcrypt unregistered [ 915.806091] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 916.042463] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 916.044676] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 916.049827] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 918.817641] Key type ._llcrypt registered [ 918.819986] Key type .llcrypt registered [ 918.998518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 919.025948] alg: No test for adler32 (adler32-zlib) [ 920.374852] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 927.076313] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 15:51:49 (1775764309) [ 927.216115] LNet: 52170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.226632] LNetError: 52170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 927.253252] LNet: Removed LNI 10.1.2.3@tcp [ 928.047513] Key type .llcrypt unregistered [ 928.053812] Key type ._llcrypt unregistered [ 934.128801] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 934.426371] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 934.431493] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 934.445603] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 936.167184] Key type ._llcrypt registered [ 936.170504] Key type .llcrypt registered [ 936.398046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 936.430656] alg: No test for adler32 (adler32-zlib) [ 937.496468] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 945.234313] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 15:52:07 (1775764327) [ 945.568191] LNet: 53062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.579333] LNetError: 53062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 945.599135] LNet: Removed LNI 10.1.2.3@tcp [ 946.555610] Key type .llcrypt unregistered [ 946.567506] Key type ._llcrypt unregistered [ 953.374380] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 953.593494] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 953.596177] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 953.601188] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 956.371986] Key type ._llcrypt registered [ 956.375023] Key type .llcrypt registered [ 956.569111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 956.614972] alg: No test for adler32 (adler32-zlib) [ 957.603641] LNet: 53363:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 957.617454] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 964.511869] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 15:52:27 (1775764347) [ 964.609448] LNet: 53948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.615326] LNetError: 53948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 964.637941] LNet: Removed LNI 10.1.2.3@tcp [ 965.274277] Key type .llcrypt unregistered [ 965.277627] Key type ._llcrypt unregistered [ 971.126119] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 971.434168] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 971.440775] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 971.454460] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 974.270437] Key type ._llcrypt registered [ 974.274230] Key type .llcrypt registered [ 974.483604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 974.499649] alg: No test for adler32 (adler32-zlib) [ 975.402480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.496740] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 983.950698] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 15:52:46 (1775764366) [ 984.512448] LNet: 55184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.523087] LNetError: 55184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 984.535386] LNet: Removed LNI 10.1.2.3@tcp [ 985.267442] Key type .llcrypt unregistered [ 985.270752] Key type ._llcrypt unregistered [ 992.124521] Key type ._llcrypt registered [ 992.134399] Key type .llcrypt registered [ 992.547845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 992.573759] alg: No test for adler32 (adler32-zlib) [ 993.697428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 993.722663] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1001.694751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.145@tcp [ 1002.679098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_interrupt -r 1 [ 1002.753965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_interrupt -r 1 [ 1002.808540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1002.829421] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1003.044904] LNet: 1 local NIs in recovery (showing 1): 192.168.204.45@tcp [ 1003.154881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1003.231117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1003.294835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1003.369258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1003.478393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1003.548976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1003.619492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1003.678495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1003.718244] LNet: 55621:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1003.748541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1003.813507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1003.865733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1003.943253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1004.015548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1004.062223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1004.138480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1005.224931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_dropped -r 1 [ 1005.280345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_dropped -r 1 [ 1005.333584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1005.361496] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1005.379397] LNet: Skipped 1 previous similar message [ 1005.723504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1005.795549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1005.854484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1005.929249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1005.996210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1006.084635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1006.128899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1006.175104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1006.212406] LNet: 55731:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1006.258750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1006.341585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1006.406834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1006.450337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1006.506482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1006.553305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1006.610359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1007.372875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_aborted -r 1 [ 1007.435691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_aborted -r 1 [ 1007.483902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1007.512757] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1007.746514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1007.776525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1007.832249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1007.883658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1007.937687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1007.985320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1008.052801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1008.093443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1008.124854] LNet: 55840:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1008.157769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1008.209639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1008.252055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1008.297612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1008.345708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1008.438952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1008.498486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1009.088690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_no_route -r 1 [ 1009.148172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_no_route -r 1 [ 1009.200983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1009.531551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1009.561647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1009.593645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1009.626476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1009.659509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1009.699460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1009.761611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1009.806292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1009.845538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1009.893985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1009.921297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1009.945935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1009.974434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1010.040548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1010.073947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1010.930438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_timeout -r 1 [ 1010.989518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_timeout -r 1 [ 1011.067731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1011.120198] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1011.128432] LNet: Skipped 1 previous similar message [ 1011.433444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1011.456788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1011.481618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1011.517638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1011.550441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1011.578766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1011.626492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1011.672814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1011.704174] LNet: 56059:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1011.714745] LNet: 56059:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1011.741835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1011.787166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1011.812380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1011.841125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1011.879831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1011.909923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1011.940912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1012.493909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_error -r 1 [ 1012.540725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_error -r 1 [ 1012.599323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1012.846964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1012.881530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1012.916448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1012.962307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1012.998281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1013.029408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1013.064521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1013.101406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1013.134519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1013.165426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1013.196904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1013.224942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1013.265705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1013.318871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1013.355537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1013.762038] LNet: 56209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.775830] LNetError: 56209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.802046] LNet: Removed LNI 192.168.204.45@tcp [ 1014.164925] Key type .llcrypt unregistered [ 1014.166885] Key type ._llcrypt unregistered [ 1030.323333] Key type ._llcrypt registered [ 1030.325622] Key type .llcrypt registered [ 1031.575454] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 15:53:34 (1775764414) [ 1032.381371] Key type .llcrypt unregistered [ 1032.384729] Key type ._llcrypt unregistered [ 1038.044478] Key type ._llcrypt registered [ 1038.046945] Key type .llcrypt registered [ 1038.311491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1038.336114] alg: No test for adler32 (adler32-zlib) [ 1039.306944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1039.350465] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1046.718227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.145@tcp [ 1057.569341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.620722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1057.653877] LNet: Added LNI 192.168.204.45@tcp1 [8/256/0/180] [ 1059.305679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_interrupt -r 1 [ 1059.334772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e local_interrupt -r 1 [ 1059.364833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_interrupt -r 1 [ 1059.393530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e local_interrupt -r 1 [ 1059.437987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e local_interrupt -r 1 [ 1059.468825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e local_interrupt -r 1 [ 1059.519398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e local_interrupt -r 1 [ 1059.564428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e local_interrupt -r 1 [ 1059.621135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1059.643735] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1059.649696] LNet: 1 local NIs in recovery (showing 1): 192.168.204.45@tcp [ 1059.874771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1059.924476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1059.950856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1060.000273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1060.034648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1060.074899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1060.126923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1060.173909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1060.197864] LNet: 57685:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1060.214751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1060.237504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1060.260123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1060.283422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1060.309506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1060.358930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1060.405564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1060.921169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_dropped -r 1 [ 1061.003183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e local_dropped -r 1 [ 1061.054617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_dropped -r 1 [ 1061.113177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e local_dropped -r 1 [ 1061.178166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e local_dropped -r 1 [ 1061.223774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e local_dropped -r 1 [ 1061.274509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e local_dropped -r 1 [ 1061.306836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e local_dropped -r 1 [ 1061.338702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1061.357252] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1061.363235] LNet: Skipped 2 previous similar messages [ 1061.572611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1061.598983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1061.633730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1061.673593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1061.738616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1061.775676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1061.808321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1061.843396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1061.869582] LNet: 57806:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1061.891833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1061.928436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1061.967978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1062.008079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1062.051131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1062.126428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1062.182861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1062.857852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_aborted -r 1 [ 1062.890736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e local_aborted -r 1 [ 1062.928258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_aborted -r 1 [ 1062.957711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e local_aborted -r 1 [ 1062.986557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e local_aborted -r 1 [ 1063.017553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e local_aborted -r 1 [ 1063.049916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e local_aborted -r 1 [ 1063.090657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e local_aborted -r 1 [ 1063.130734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1063.150087] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1063.154314] LNet: Skipped 2 previous similar messages [ 1063.406793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1063.457654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1063.514177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1063.555795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1063.590323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1063.646434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1063.703342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1063.763482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1063.796526] LNet: 57927:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1063.837067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1063.869684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1063.900490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1063.933374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1063.974762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1064.013601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1064.061506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1064.673435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_no_route -r 1 [ 1064.710066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e local_no_route -r 1 [ 1064.764779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_no_route -r 1 [ 1064.822887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e local_no_route -r 1 [ 1064.856469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e local_no_route -r 1 [ 1064.889539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e local_no_route -r 1 [ 1064.928167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e local_no_route -r 1 [ 1064.960630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e local_no_route -r 1 [ 1064.990509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1065.252907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1065.310021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1065.352141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1065.393539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1065.445393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1065.493991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1065.535919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1065.565393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1065.627936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1065.700095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1065.751460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1065.791934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1065.829896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1065.858226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1065.902458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1066.398171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_timeout -r 1 [ 1066.431492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e local_timeout -r 1 [ 1066.468411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_timeout -r 1 [ 1066.511473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e local_timeout -r 1 [ 1066.539504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e local_timeout -r 1 [ 1066.602473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e local_timeout -r 1 [ 1066.670659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e local_timeout -r 1 [ 1066.732331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e local_timeout -r 1 [ 1066.789937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1066.827366] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1066.837784] LNet: Skipped 5 previous similar messages [ 1067.080827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1067.124248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1067.183513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1067.259723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1067.298936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1067.353274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1067.411120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1067.479167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1067.505462] LNet: 58170:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1067.513317] LNet: 58170:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1067.533460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1067.556935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1067.584985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1067.626059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1067.665958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1067.703034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1067.773083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1068.292521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_error -r 1 [ 1068.319673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e local_error -r 1 [ 1068.356723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_error -r 1 [ 1068.405829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e local_error -r 1 [ 1068.445028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e local_error -r 1 [ 1068.490301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e local_error -r 1 [ 1068.536292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e local_error -r 1 [ 1068.580279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e local_error -r 1 [ 1068.617806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1068.955880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1068.990992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1069.035608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1069.078896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1069.124404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1069.180726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1069.220781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1069.249787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1069.278413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1069.312775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1069.351919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1069.374913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1069.395698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1069.425282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1069.451841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1074.833974] LNet: 58367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.848913] LNetError: 58367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.863421] LNet: Removed LNI 192.168.204.45@tcp [ 1074.896556] LNet: Removed LNI 192.168.204.45@tcp1 [ 1075.471849] Key type .llcrypt unregistered [ 1075.477532] Key type ._llcrypt unregistered [ 1095.180852] Key type ._llcrypt registered [ 1095.186460] Key type .llcrypt registered [ 1096.581792] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 15:54:39 (1775764479) [ 1097.327078] Key type .llcrypt unregistered [ 1097.332415] Key type ._llcrypt unregistered [ 1104.062646] Key type ._llcrypt registered [ 1104.066202] Key type .llcrypt registered [ 1104.249383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1104.272056] alg: No test for adler32 (adler32-zlib) [ 1105.284962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1105.328866] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1113.606759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.145@tcp [ 1114.396584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e remote_dropped -r 1 [ 1114.451069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e remote_dropped -r 1 [ 1114.501813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1114.521257] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1114.736492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1114.769318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1114.807678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.870933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.922358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.964532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1115.009378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1115.053678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1115.078477] LNet: 59617:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1115.108962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1115.150821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.190830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.230390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.258477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.297772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.333877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1115.886710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e remote_error -r 1 [ 1115.954588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e remote_error -r 1 [ 1115.989745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1116.024919] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1116.268394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.320175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1116.352625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.388435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.438178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.496859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.549340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.594537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.633031] LNet: 59726:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1116.661593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.707527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.771754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.814807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.851964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.887585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1116.924827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1117.526364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e remote_timeout -r 1 [ 1117.580764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e remote_timeout -r 1 [ 1117.626996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1117.667723] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1117.936159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1117.979611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1118.050364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1118.100541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1118.174868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1118.238553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.288749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.362213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.382591] LNet: 59836:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1118.407424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1118.431751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.458468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1118.488324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.546607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.583765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1118.632481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1118.960751] LNet: 59877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.965940] LNetError: 59877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.986090] LNet: Removed LNI 192.168.204.45@tcp [ 1119.349909] Key type .llcrypt unregistered [ 1119.352711] Key type ._llcrypt unregistered [ 1137.508662] Key type ._llcrypt registered [ 1137.510911] Key type .llcrypt registered [ 1138.677927] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 15:55:21 (1775764521) [ 1139.575624] Key type .llcrypt unregistered [ 1139.578156] Key type ._llcrypt unregistered [ 1146.339333] Key type ._llcrypt registered [ 1146.344333] Key type .llcrypt registered [ 1146.612965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1146.635900] alg: No test for adler32 (adler32-zlib) [ 1147.612348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1147.677821] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1155.678822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.145@tcp [ 1164.952944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1164.991671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1165.015074] LNet: Added LNI 192.168.204.45@tcp1 [8/256/0/180] [ 1166.647475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e remote_dropped -r 1 [ 1166.714214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e remote_dropped -r 1 [ 1166.763099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e remote_dropped -r 1 [ 1166.816478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e remote_dropped -r 1 [ 1166.855721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e remote_dropped -r 1 [ 1166.908603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e remote_dropped -r 1 [ 1166.945637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e remote_dropped -r 1 [ 1166.993496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e remote_dropped -r 1 [ 1167.066220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1167.088780] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1167.098537] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.145@tcp [ 1167.331119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.357522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1167.402600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.445088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.481506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.534795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.565777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.608629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.634631] LNet: 61345:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1167.659813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.706995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.749538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.790506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.833454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.877032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.924391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1167.974334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.538602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e remote_error -r 1 [ 1168.577755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e remote_error -r 1 [ 1168.610732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e remote_error -r 1 [ 1168.653854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e remote_error -r 1 [ 1168.704778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e remote_error -r 1 [ 1168.761108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e remote_error -r 1 [ 1168.821482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e remote_error -r 1 [ 1168.885501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e remote_error -r 1 [ 1168.926967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1168.957171] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1168.960553] LNet: Skipped 2 previous similar messages [ 1169.227913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.261097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1169.300961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.355493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.404567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.447555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1169.483991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1169.518736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1169.543905] LNet: 61468:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1169.559390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1169.595851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1169.627522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1169.658922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.690252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.745555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.778604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1169.831485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.377732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e remote_timeout -r 1 [ 1170.418807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e remote_timeout -r 1 [ 1170.445887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e remote_timeout -r 1 [ 1170.476878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e remote_timeout -r 1 [ 1170.506946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e remote_timeout -r 1 [ 1170.538782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e remote_timeout -r 1 [ 1170.566619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e remote_timeout -r 1 [ 1170.611204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e remote_timeout -r 1 [ 1170.664925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.145@tcp [ 1170.699089] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1170.701991] LNet: Skipped 1 previous similar message [ 1170.965301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.017244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1171.082228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.125222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.202482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.256367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.287508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.317800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.334792] LNet: 61591:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.376106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.453330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.491534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.521173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.557591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.604449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.646967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1171.688328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.771833] LNet: 61669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1176.783629] LNetError: 61669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1176.790927] LNet: Removed LNI 192.168.204.45@tcp [ 1176.821981] LNet: Removed LNI 192.168.204.45@tcp1 [ 1177.287627] Key type .llcrypt unregistered [ 1177.301195] Key type ._llcrypt unregistered [ 1195.215683] Key type ._llcrypt registered [ 1195.218844] Key type .llcrypt registered [ 1196.756418] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 15:56:19 (1775764579) [ 1197.579736] Key type .llcrypt unregistered [ 1197.581527] Key type ._llcrypt unregistered [ 1205.548361] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1205.778180] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1205.794546] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1205.796621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1208.434133] Key type ._llcrypt registered [ 1208.441263] Key type .llcrypt registered [ 1208.617578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1208.630849] alg: No test for adler32 (adler32-zlib) [ 1209.668634] LNet: 62746:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1209.702550] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1209.760299] LNet: 62760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.772536] LNetError: 62760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.793474] LNet: Removed LNI 192.168.204.45@tcp [ 1210.539410] Key type .llcrypt unregistered [ 1210.544593] Key type ._llcrypt unregistered [ 1216.794235] Key type ._llcrypt registered [ 1216.797549] Key type .llcrypt registered [ 1217.075132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1217.104126] alg: No test for adler32 (adler32-zlib) [ 1218.071451] LNet: 63019:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1218.090791] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1218.094098] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1218.166054] LNet: 63033:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1218.174462] LNetError: 63033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1218.191291] LNet: Removed LNI 192.168.204.45@tcp [ 1218.215293] LNet: Removed LNI 10.1.2.3@tcp1 [ 1218.893564] Key type .llcrypt unregistered [ 1218.900313] Key type ._llcrypt unregistered [ 1225.571813] Key type ._llcrypt registered [ 1225.573343] Key type .llcrypt registered [ 1225.788324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1225.818026] alg: No test for adler32 (adler32-zlib) [ 1226.815906] LNet: 63292:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1226.838907] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1226.842117] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1226.917690] LNet: 63306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.924886] LNetError: 63306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.932495] LNet: Removed LNI 192.168.204.45@tcp [ 1226.941966] LNet: Removed LNI 10.1.2.3@tcp [ 1227.648469] Key type .llcrypt unregistered [ 1227.652429] Key type ._llcrypt unregistered [ 1234.459088] Key type ._llcrypt registered [ 1234.461270] Key type .llcrypt registered [ 1234.722218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1234.743392] alg: No test for adler32 (adler32-zlib) [ 1235.740961] LNet: 63566:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1235.765642] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1235.772158] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1235.845366] LNet: 63580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1235.853184] LNetError: 63580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1235.861564] LNet: Removed LNI 192.168.204.45@tcp [ 1235.875083] LNet: Removed LNI 10.1.2.3@tcp [ 1236.775588] Key type .llcrypt unregistered [ 1236.778106] Key type ._llcrypt unregistered [ 1243.331728] Key type ._llcrypt registered [ 1243.338168] Key type .llcrypt registered [ 1243.686587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1243.710054] alg: No test for adler32 (adler32-zlib) [ 1244.678273] LNet: 63845:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1244.688092] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1244.690261] LNetError: 63845:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1244.696136] LNetError: Error -100 starting up LNI tcp [ 1244.724561] LNet: Removed LNI 192.168.204.45@tcp [ 1245.484516] Key type .llcrypt unregistered [ 1245.486115] Key type ._llcrypt unregistered [ 1256.426043] Key type ._llcrypt registered [ 1256.427363] Key type .llcrypt registered [ 1257.754954] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 15:57:20 (1775764640) [ 1258.868369] Key type .llcrypt unregistered [ 1258.879298] Key type ._llcrypt unregistered [ 1265.714697] Key type ._llcrypt registered [ 1265.717683] Key type .llcrypt registered [ 1265.947969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1265.967510] alg: No test for adler32 (adler32-zlib) [ 1266.932355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1266.986214] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1274.625925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.145@tcp [ 1275.488870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e network_timeout -r 1 [ 1275.539467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e network_timeout -r 1 [ 1275.578597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.145@tcp [ 1275.600892] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1275.870392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1275.920599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1275.960646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1276.005405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1276.052988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1276.089857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1276.130139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1276.194788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1276.234713] LNet: 65101:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1276.257326] LNet: 1 local NIs in recovery (showing 1): 192.168.204.45@tcp [ 1276.276959] LNet: There was an unexpected network error while writing to 192.168.204.45: rc = -22 [ 1276.294506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1276.354897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1276.407315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1276.432059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1276.477230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.522651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1276.822634] LNet: 65140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1276.845384] LNetError: 65140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1276.861366] LNet: Removed LNI 192.168.204.45@tcp [ 1277.458620] Key type .llcrypt unregistered [ 1277.460551] Key type ._llcrypt unregistered [ 1296.289900] Key type ._llcrypt registered [ 1296.291645] Key type .llcrypt registered [ 1296.684406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1296.705326] alg: No test for adler32 (adler32-zlib) [ 1297.557370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1297.572680] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1305.861944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.145@tcp [ 1316.986931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1317.091305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1317.121718] LNet: Added LNI 192.168.204.45@tcp1 [8/256/0/180] [ 1319.248222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e network_timeout -r 1 [ 1319.290868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e network_timeout -r 1 [ 1319.347587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e network_timeout -r 1 [ 1319.416816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e network_timeout -r 1 [ 1319.481961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e network_timeout -r 1 [ 1319.522858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e network_timeout -r 1 [ 1319.574374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e network_timeout -r 1 [ 1319.659091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e network_timeout -r 1 [ 1319.739770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.145@tcp [ 1319.812517] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1320.104980] LNet: 1 local NIs in recovery (showing 1): 192.168.204.45@tcp [ 1320.136415] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.145@tcp [ 1320.224944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1320.283787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1320.325649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1320.368035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1320.408783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1320.488473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1320.536670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1320.602936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1320.639202] LNet: 66020:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1320.663326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1320.707741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1320.746324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1320.856893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1320.942112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1321.000034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1326.089182] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1326.100065] LNet: Skipped 2 previous similar messages [ 1326.463918] LNet: 66094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.474639] LNetError: 66094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.519932] LNet: Removed LNI 192.168.204.45@tcp [ 1327.526783] LNetError: 66094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.548731] LNet: Removed LNI 192.168.204.45@tcp1 [ 1328.065473] Key type .llcrypt unregistered [ 1328.067865] Key type ._llcrypt unregistered [ 1346.773976] Key type ._llcrypt registered [ 1346.781699] Key type .llcrypt registered [ 1348.330589] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 15:58:50 (1775764730) [ 1348.987285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1349.008759] alg: No test for adler32 (adler32-zlib) [ 1349.978552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1350.074839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1350.108597] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1350.914790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1350.954511] LNet: Added LNI 192.168.204.45@tcp1 [8/256/0/180] [ 1351.985620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 1352.094655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1352.149937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1352.226306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1352.278241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1352.331909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1352.379677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.45@tcp [ 1352.404638] LNetError: 67195:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.45@tcp to recovery [ 1353.119463] LNet: 1 local NIs in recovery (showing 1): 192.168.204.45@tcp [ 1353.125209] LNet: There was an unexpected network error while writing to 192.168.204.45: rc = -22 [ 1355.167364] LNet: There was an unexpected network error while writing to 192.168.204.45: rc = -22 [ 1355.952946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1359.265098] LNet: There was an unexpected network error while writing to 192.168.204.45: rc = -22 [ 1359.573633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1359.629753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1359.752426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1359.779293] LNet: 67302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.787962] LNetError: 67302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.799197] LNet: Removed LNI 192.168.204.45@tcp [ 1360.801990] LNetError: 67302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.823394] LNet: Removed LNI 192.168.204.45@tcp1 [ 1360.872636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.950276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1360.987492] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1361.888358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1361.908102] LNet: Added LNI 192.168.204.45@tcp1 [8/256/0/180] [ 1363.005240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 1363.080511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1363.153841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1363.214345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1363.254740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1363.315447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1363.382132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1363.430134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.45@tcp [ 1363.451826] LNetError: 67574:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.45@tcp to recovery [ 1364.000083] LNet: There was an unexpected network error while writing to 192.168.204.45: rc = -22 [ 1365.930242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1369.120528] LNet: There was an unexpected network error while writing to 192.168.204.45: rc = -22 [ 1369.127758] LNet: Skipped 1 previous similar message [ 1374.010717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1374.060436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1374.117344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1381.026435] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 15:59:23 (1775764763) [ 1381.163359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1381.201446] LNet: 68277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1381.217372] LNetError: 68277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1381.227731] LNet: Removed LNI 192.168.204.45@tcp [ 1381.267719] LNet: Removed LNI 192.168.204.45@tcp1 [ 1381.306747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.348915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1381.371448] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1382.218549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1382.245644] LNet: Added LNI 192.168.204.45@tcp1 [8/256/0/180] [ 1383.310283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 1383.412425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1383.451784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1383.512457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1383.589202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1383.637472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1383.723782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.45@tcp --health 0 [ 1383.880539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1384.417165] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.45@tcp [ 1384.420092] LNet: There was an unexpected network error while writing to 192.168.204.45: rc = -22 [ 1384.435869] LNet: Skipped 1 previous similar message [ 1399.613106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1399.827130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1399.878669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1399.931129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.45@tcp --health 500 [ 1400.079629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1406.801846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1406.897341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1406.922685] LNet: 68773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.938347] LNetError: 68773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.942683] LNetError: 68773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1406.951357] LNet: Removed LNI 192.168.204.45@tcp [ 1406.961974] LNet: Removed LNI 192.168.204.45@tcp1 [ 1407.006573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1407.109240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1407.173920] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1408.008741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1408.047396] LNet: Added LNI 192.168.204.45@tcp1 [8/256/0/180] [ 1409.132548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 1409.181611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1409.246375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1409.283218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1409.313746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1409.345292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1409.381842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1409.409636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.45@tcp --health 0 [ 1410.143707] LNet: There was an unexpected network error while writing to 192.168.204.45: rc = -22 [ 1410.148994] LNet: Skipped 3 previous similar messages [ 1410.812190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1421.177248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1421.207600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1421.238534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1428.960620] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 16:00:11 (1775764811) [ 1429.442816] LNet: 69819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1429.447573] LNetError: 69819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1429.450808] LNetError: 69819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1430.496641] LNet: Removed LNI 192.168.204.45@tcp [ 1430.511617] LNet: Removed LNI 192.168.204.45@tcp1 [ 1431.104443] Key type .llcrypt unregistered [ 1431.107216] Key type ._llcrypt unregistered [ 1436.745255] Key type ._llcrypt registered [ 1436.749747] Key type .llcrypt registered [ 1436.943422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1436.951873] alg: No test for adler32 (adler32-zlib) [ 1437.899648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1437.926447] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1447.279808] Lustre: DEBUG MARKER: Initial discovery [ 1447.640959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.145@tcp [ 1449.967531] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1450.129594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1450.193160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.145@tcp [ 1450.251454] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1450.281184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1450.332880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.145@tcp -r 1 -m GET -l 3 [ 1450.384663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.145@tcp -r 1 -m GET -e local_error [ 1450.424461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.145@tcp -r 1 -m PUT -l 6 [ 1451.231448] LNet: 1 local NIs in recovery (showing 1): 192.168.204.45@tcp [ 1451.895552] Lustre: DEBUG MARKER: Force oleg445-client.virtnet to discover 192.168.204.145@tcp (in background) [ 1452.112589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.145@tcp [ 1453.449687] Lustre: DEBUG MARKER: Force oleg445-server to discover 192.168.204.45@tcp [ 1455.200891] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1455.594349] Lustre: DEBUG MARKER: Wait for 70583 [ 1457.164393] Lustre: DEBUG MARKER: Finished wait on 70583 [ 1457.459587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1457.518726] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1457.548813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1457.863977] LNet: 71031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.872512] LNetError: 71031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.886120] LNet: Removed LNI 192.168.204.45@tcp [ 1458.269578] Key type .llcrypt unregistered [ 1458.274513] Key type ._llcrypt unregistered [ 1475.052942] Key type ._llcrypt registered [ 1475.054555] Key type .llcrypt registered [ 1476.712560] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 16:00:59 (1775764859) [ 1477.629150] Key type .llcrypt unregistered [ 1477.637216] Key type ._llcrypt unregistered [ 1484.006911] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1484.090035] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1484.096053] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1484.105762] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1485.643262] Key type ._llcrypt registered [ 1485.646219] Key type .llcrypt registered [ 1485.819442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1485.845597] alg: No test for adler32 (adler32-zlib) [ 1486.828775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.890363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1486.906325] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1487.615928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1487.632167] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1495.943767] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 16:01:18 (1775764878) [ 1496.036382] LNet: 73104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1496.041715] LNetError: 73104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1496.053928] LNet: Removed LNI 192.168.204.45@tcp [ 1496.070266] LNet: Removed LNI 10.1.2.3@tcp [ 1496.823375] Key type .llcrypt unregistered [ 1496.828318] Key type ._llcrypt unregistered [ 1504.162989] Key type ._llcrypt registered [ 1504.172328] Key type .llcrypt registered [ 1504.478983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1504.499858] alg: No test for adler32 (adler32-zlib) [ 1505.580545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1505.684102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1505.718891] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1506.681914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1506.705254] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1507.894476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.45@tcp [ 1508.959474] LNetError: Unexpected error -22 connecting to 192.168.204.45@tcp at host 192.168.204.45:988 [ 1515.568583] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 16:01:38 (1775764898) [ 1515.912473] LNet: 74243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1515.930537] LNetError: 74243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1515.936875] LNet: Removed LNI 192.168.204.45@tcp [ 1515.961249] LNet: Removed LNI 10.1.2.3@tcp [ 1516.682694] Key type .llcrypt unregistered [ 1516.688088] Key type ._llcrypt unregistered [ 1522.322704] Key type ._llcrypt registered [ 1522.325780] Key type .llcrypt registered [ 1522.615806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1522.644370] alg: No test for adler32 (adler32-zlib) [ 1523.632706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1523.683836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1523.733724] LNet: Added LNI 192.168.204.45@tcp1 [8/256/0/180] [ 1524.415989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1524.434773] LNet: Added LNI 192.168.204.45@tcp2 [8/256/0/180] [ 1525.326100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.45@tcp1 --nid 192.168.204.45@tcp2 [ 1529.804984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.45@tcp1 192.168.204.45@tcp2 [ 1529.825709] LNetError: 75079:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.45@tcp1 to 192.168.204.45@tcp2 - Route table may be misconfigured [ 1529.856475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.45@tcp2 192.168.204.45@tcp1 [ 1535.437285] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 16:01:58 (1775764918) [ 1535.711386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.750650] LNet: 75652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.760085] LNetError: 75652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.767484] LNet: Removed LNI 192.168.204.45@tcp1 [ 1536.807533] LNet: Removed LNI 192.168.204.45@tcp2 [ 1536.853966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.918309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1536.941703] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1537.458963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1537.475660] LNet: Added LNI 192.168.204.45@tcp1 [8/256/0/180] [ 1538.342593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 1538.395913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.45@tcp 192.168.204.45@tcp [ 1538.420415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.45@tcp1 192.168.204.45@tcp1 [ 1538.459812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.45@tcp -d 192.168.204.45@tcp -e network_timeout [ 1538.492897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -e network_timeout [ 1538.522406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -e network_timeout [ 1538.546695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -e network_timeout [ 1538.570821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1538.589296] LNet: There was an unexpected network error while writing to 192.168.204.45: rc = -22 [ 1538.607542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1538.686632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1538.761406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1538.975462] LNet: 1 local NIs in recovery (showing 1): 192.168.204.45@tcp [ 1545.751732] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 16:02:08 (1775764928) [ 1546.286374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.313365] LNet: 76514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.320709] LNetError: 76514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.329444] LNetError: 76514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1546.338696] LNet: Removed LNI 192.168.204.45@tcp [ 1547.388593] LNet: Removed LNI 192.168.204.45@tcp1 [ 1547.454912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.642078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1548.027497] LNet: 76551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.536470] Key type .llcrypt unregistered [ 1548.540538] Key type ._llcrypt unregistered [ 1564.215853] Key type ._llcrypt registered [ 1564.222366] Key type .llcrypt registered [ 1566.369274] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 16:02:28 (1775764948) [ 1567.146928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1567.166634] alg: No test for adler32 (adler32-zlib) [ 1568.164237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.842459] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1571.634705] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 16:02:34 (1775764954) [ 1571.830123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1571.863033] LNet: 77650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.885182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.935146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1571.959333] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1572.747100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1572.768880] LNet: Added LNI 192.168.204.45@tcp1 [8/256/0/180] [ 1573.946339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1573.994915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp1 [ 1574.056762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp1 [ 1580.201919] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 16:02:43 (1775764963) [ 1582.517415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg445-server [ 1584.169380] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 16:02:47 (1775764967) [ 1586.141368] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg445-server [ 1588.100255] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 16:02:50 (1775764970) [ 1590.098368] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg445-server [ 1592.109981] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 16:02:54 (1775764974) [ 1594.793832] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg445-server [ 1597.278193] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 16:02:59 (1775764979) [ 1599.640688] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg445-server [ 1601.917144] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 16:03:04 (1775764984) [ 1604.075153] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg445-server [ 1605.771987] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 16:03:08 (1775764988) [ 1607.419415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg445-server [ 1609.355657] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 16:03:12 (1775764992) [ 1611.236775] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg445-server [ 1612.959345] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 16:03:16 (1775764996) [ 1615.053151] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg445-server [ 1616.696426] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 16:03:19 (1775764999) [ 1616.903090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1616.916607] LNet: 81528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.922923] LNetError: 81528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.931955] LNet: Removed LNI 192.168.204.45@tcp [ 1617.965452] LNet: Removed LNI 192.168.204.45@tcp1 [ 1618.021075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.077066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1618.094990] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1618.780539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1618.856355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1619.036495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.058831] LNet: 81676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1619.073779] LNetError: 81676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1619.089485] LNetError: 81676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1619.123504] LNet: Removed LNI 192.168.204.45@tcp [ 1619.155460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.247524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1619.273033] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1620.202836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1620.405924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.432044] LNetError: 81821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1620.450439] LNet: Removed LNI 192.168.204.45@tcp [ 1620.468655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.512903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1620.538660] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1621.314463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1621.390983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1621.567101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.587660] LNet: 81968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.594578] LNet: 81968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1622.636168] LNet: Removed LNI 192.168.204.45@tcp [ 1622.666374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.728208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1622.751768] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1623.526477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1623.670118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.684869] LNetError: 82114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.690862] LNetError: 82114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1623.708895] LNet: Removed LNI 192.168.204.45@tcp [ 1623.744341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.799878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1623.828959] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1624.810219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1624.893094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1625.157231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.214609] LNet: Removed LNI 192.168.204.45@tcp [ 1626.237189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.339749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1626.376160] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1627.372484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1627.618947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.641127] LNet: 82406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.647983] LNet: 82406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1628.711445] LNet: Removed LNI 192.168.204.45@tcp [ 1628.770641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.848805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1628.896265] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1629.569243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1629.648301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1631.092496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.126560] LNetError: 82563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.131911] LNetError: 82563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1632.171356] LNet: Removed LNI 192.168.204.45@tcp [ 1632.187478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.274897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1632.324659] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1633.065413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1633.331368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.405826] LNet: Removed LNI 192.168.204.45@tcp [ 1634.459523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.574350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1634.599730] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1635.552919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1635.659195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1635.975532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.016894] LNet: 82855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.021712] LNet: 82855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1637.095260] LNet: Removed LNI 192.168.204.45@tcp [ 1637.141886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.232542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1637.273275] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1638.258412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1638.582462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.654238] LNet: Removed LNI 192.168.204.45@tcp [ 1639.721460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.789662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1639.827196] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1640.694585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1640.770609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1641.162497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.182157] LNetError: 83148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1641.192961] LNetError: 83148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1641.224984] LNet: Removed LNI 192.168.204.45@tcp [ 1641.262623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.317384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1641.343754] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1642.318518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1642.671605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1643.751868] LNet: Removed LNI 192.168.204.45@tcp [ 1643.797312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.883019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1643.914597] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1644.714486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1644.771211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.813083] LNet: Removed LNI 192.168.204.45@tcp [ 1644.834545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.922724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1644.952382] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1645.726615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1653.050194] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 16:03:55 (1775765035) [ 1653.127105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.162159] LNet: 84125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.169595] LNet: 84125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1653.189236] LNet: Removed LNI 192.168.204.45@tcp [ 1653.222740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.297336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1653.338491] LNet: Added LNI 192.168.204.45@tcp231 [8/256/0/180] [ 1655.009863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.061107] LNet: Removed LNI 192.168.204.45@tcp231 [ 1655.095768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.152870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1655.182223] LNet: Added LNI 192.168.204.45@tcp231 [8/256/0/180] [ 1655.196808] LNet: 84277:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1655.736106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1655.784443] LNet: Removed LNI 192.168.204.45@tcp231 [ 1655.829933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1655.870219] LNet: Added LNI 192.168.204.45@tcp231 [8/256/0/180] [ 1657.047771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1657.078705] LNet: Removed LNI 192.168.204.45@tcp231 [ 1657.095745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.169140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1657.206104] LNet: Added LNI 192.168.204.45@tcp231 [8/256/0/0] [ 1657.210587] LNet: 84429:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1657.669707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1657.722335] LNet: Removed LNI 192.168.204.45@tcp231 [ 1657.755727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1657.807993] LNet: Added LNI 192.168.204.45@tcp231 [8/256/0/0] [ 1659.147581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1659.165474] LNetError: 84573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1659.174741] LNetError: 84573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1659.186334] LNet: Removed LNI 192.168.204.45@tcp231 [ 1659.247466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.362491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1659.421651] LNet: Added LNI 192.168.204.45@tcp231 [8/256/0/60] [ 1659.427939] LNet: 84581:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1660.042941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1660.124500] LNet: Removed LNI 192.168.204.45@tcp231 [ 1660.145863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1660.185149] LNet: Added LNI 192.168.204.45@tcp231 [8/256/0/60] [ 1661.886609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1661.924158] LNet: Removed LNI 192.168.204.45@tcp231 [ 1661.963057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.062791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1662.112103] LNet: Added LNI 192.168.204.45@tcp231 [8/256/0/180] [ 1662.118704] LNet: 84732:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1662.580937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1662.618318] LNet: Removed LNI 192.168.204.45@tcp231 [ 1662.635451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1662.670948] LNet: Added LNI 192.168.204.45@tcp231 [8/256/0/180] [ 1670.758222] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 16:04:13 (1775765053) [ 1671.120649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1671.165782] LNet: Removed LNI 192.168.204.45@tcp231 [ 1671.199994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1671.336982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1671.383407] LNet: Added LNI 192.168.204.45@tcp232 [8/256/0/180] [ 1672.154559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1672.202847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1672.232407] LNet: Removed LNI 192.168.204.45@tcp232 [ 1672.260447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1672.304702] LNet: Added LNI 192.168.204.45@tcp232 [8/256/0/180] [ 1673.095662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1673.147375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1673.173892] LNet: Removed LNI 192.168.204.45@tcp232 [ 1673.191790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1673.240722] LNet: Added LNI 192.168.204.45@tcp232 [8/256/0/180] [ 1673.931105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1673.995789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1674.030837] LNet: Removed LNI 192.168.204.45@tcp232 [ 1674.055876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1674.079070] LNet: Added LNI 192.168.204.45@tcp232 [8/256/0/180] [ 1674.943097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1674.991941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1675.025896] LNet: Removed LNI 192.168.204.45@tcp232 [ 1675.041797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1681.268777] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 16:04:24 (1775765064) [ 1682.345863] Key type .llcrypt unregistered [ 1682.348521] Key type ._llcrypt unregistered [ 1688.563236] Key type ._llcrypt registered [ 1688.564981] Key type .llcrypt registered [ 1688.762279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1688.784872] alg: No test for adler32 (adler32-zlib) [ 1689.698285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1689.727772] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1697.319782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.145@tcp [ 1707.233718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.277245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1707.314987] LNet: Added LNI 192.168.204.45@tcp1 [8/256/0/180] [ 1708.843832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_interrupt -r 2 [ 1708.879753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e local_interrupt -r 2 [ 1708.921326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_interrupt -r 2 [ 1708.954766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e local_interrupt -r 2 [ 1708.983593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e local_interrupt -r 2 [ 1709.018663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e local_interrupt -r 2 [ 1709.057052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e local_interrupt -r 2 [ 1709.118175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e local_interrupt -r 2 [ 1709.171705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.45@tcp 192.168.204.145@tcp [ 1709.199950] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1709.214023] LNet: 1 local NIs in recovery (showing 1): 192.168.204.45@tcp [ 1709.471173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1709.508728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1709.554124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1709.585097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1709.646104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1709.685702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1709.710933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1709.737154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1709.752600] LNet: 87232:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1709.767670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1709.795161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1709.831299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1709.858769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1709.920519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1709.969587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1710.540624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_dropped -r 2 [ 1710.592933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e local_dropped -r 2 [ 1710.639180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_dropped -r 2 [ 1710.693610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e local_dropped -r 2 [ 1710.749788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e local_dropped -r 2 [ 1710.811268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e local_dropped -r 2 [ 1710.865062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e local_dropped -r 2 [ 1710.912524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e local_dropped -r 2 [ 1710.987389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.45@tcp 192.168.204.145@tcp [ 1711.021725] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1711.311323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1711.383408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1711.425153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1711.464190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1711.491680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1711.536822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1711.588649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1711.620856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1711.643420] LNet: 87352:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1711.668429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1711.706509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1711.763297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1711.801813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1711.857685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1711.903675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1712.550994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_aborted -r 2 [ 1712.620416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e local_aborted -r 2 [ 1712.695757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_aborted -r 2 [ 1712.735612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e local_aborted -r 2 [ 1712.781521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e local_aborted -r 2 [ 1712.837400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e local_aborted -r 2 [ 1712.913154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e local_aborted -r 2 [ 1712.986313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e local_aborted -r 2 [ 1713.047321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.45@tcp 192.168.204.145@tcp [ 1713.070263] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1713.420802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1713.477349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1713.517582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1713.560888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1713.607840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1713.659884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1713.732403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1713.785939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1713.815222] LNet: 87473:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1713.851657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1713.898961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1713.923959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1713.955583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1713.998306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1714.064156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1714.686444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_no_route -r 2 [ 1714.725634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e local_no_route -r 2 [ 1714.774293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_no_route -r 2 [ 1714.830310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e local_no_route -r 2 [ 1714.869996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e local_no_route -r 2 [ 1714.922761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e local_no_route -r 2 [ 1714.968390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e local_no_route -r 2 [ 1715.026526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e local_no_route -r 2 [ 1715.096437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.45@tcp 192.168.204.145@tcp [ 1715.134655] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1715.419239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1715.449477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1715.480930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1715.514486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1715.558048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1715.613601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1715.671377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1715.713044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1715.744651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1715.780685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1715.808584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1715.832219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1715.857280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1715.884372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1716.547150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e local_timeout -r 2 [ 1716.621134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e local_timeout -r 2 [ 1716.693971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e local_timeout -r 2 [ 1716.791359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e local_timeout -r 2 [ 1716.894941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e local_timeout -r 2 [ 1716.967948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e local_timeout -r 2 [ 1717.058609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e local_timeout -r 2 [ 1717.108422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e local_timeout -r 2 [ 1717.156890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.45@tcp 192.168.204.145@tcp [ 1717.518495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1717.560640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1717.616993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1717.647991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1717.710435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1717.773752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1717.836832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1717.911801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1717.947110] LNet: 87713:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1717.958400] LNet: 87713:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1717.991188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1718.063291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1718.106589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1718.140384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1718.169166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1718.211083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1718.907561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp -m GET -e remote_dropped -r 2 [ 1719.006983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.145@tcp1 -m GET -e remote_dropped -r 2 [ 1719.059156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp -m GET -e remote_dropped -r 2 [ 1719.104847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.45@tcp -d 192.168.204.45@tcp1 -m GET -e remote_dropped -r 2 [ 1719.161634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp -m GET -e remote_dropped -r 2 [ 1719.221974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.145@tcp1 -m GET -e remote_dropped -r 2 [ 1719.274309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp -m GET -e remote_dropped -r 2 [ 1719.334895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.45@tcp1 -d 192.168.204.45@tcp1 -m GET -e remote_dropped -r 2 [ 1719.393757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.45@tcp 192.168.204.145@tcp [ 1719.432317] LNet: There was an unexpected network error while writing to 192.168.204.145: rc = -22 [ 1719.440806] LNet: Skipped 1 previous similar message [ 1719.446186] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.145@tcp [ 1719.706803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1719.750391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1719.799825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1719.845651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1719.910379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1719.943387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1720.018235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1720.063329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1720.111148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1720.154799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1720.195949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1720.241243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1720.292541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1720.338448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1726.783925] LNet: 87907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.790872] LNetError: 87907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.802132] LNet: Removed LNI 192.168.204.45@tcp [ 1726.820504] LNet: Removed LNI 192.168.204.45@tcp1 [ 1727.244559] Key type .llcrypt unregistered [ 1727.248700] Key type ._llcrypt unregistered [ 1743.689043] Key type ._llcrypt registered [ 1743.692463] Key type .llcrypt registered [ 1745.133325] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 16:05:27 (1775765127) [ 1745.883387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1745.919084] alg: No test for adler32 (adler32-zlib) [ 1746.930244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.995616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1747.035762] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1748.169437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1748.240555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.45@tcp [ 1748.269469] LNetError: 88887:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.45@tcp to recovery [ 1748.352100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.45@tcp [ 1748.442945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1755.428865] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 16:05:38 (1775765138) [ 1755.581556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.619609] LNet: 89474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.627906] LNetError: 89474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.645552] LNet: Removed LNI 192.168.204.45@tcp [ 1755.670145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.719777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1755.776724] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1756.540696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1756.566273] LNet: Added LNI 192.168.204.45@tcp1 [8/256/0/180] [ 1757.694291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.45@tcp [ 1757.713161] LNetError: 89742:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.45@tcp to recovery [ 1757.732839] LNet: 1 local NIs in recovery (showing 1): 192.168.204.45@tcp [ 1757.733793] LNet: There was an unexpected network error while writing to 192.168.204.45: rc = -22 [ 1757.860885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.45@tcp [ 1766.585307] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 16:05:49 (1775765149) [ 1768.835129] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg445-server [ 1770.792490] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 16:05:53 (1775765153) [ 1772.603271] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg445-server [ 1774.222547] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 16:05:57 (1775765157) [ 1776.198177] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg445-server [ 1777.865185] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 16:06:00 (1775765160) [ 1777.977547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.013225] LNet: 91357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.022676] LNetError: 91357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.031871] LNet: Removed LNI 192.168.204.45@tcp [ 1778.046827] LNet: Removed LNI 192.168.204.45@tcp1 [ 1778.067180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.116515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1778.143893] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1778.965462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1779.028258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1779.079696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1779.170559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1779.206415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1779.292790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1786.196823] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 16:06:08 (1775765168) [ 1786.466587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.501589] LNet: 92089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.509796] LNetError: 92089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.518027] LNetError: 92089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1786.545396] LNet: Removed LNI 192.168.204.45@tcp [ 1786.562529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.613990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1786.651061] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1793.776746] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 16:06:16 (1775765176) [ 1794.002464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.019980] LNet: 92793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1794.026898] LNetError: 92793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1794.040950] LNet: Removed LNI 192.168.204.45@tcp [ 1794.058620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.137909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1794.176497] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1800.967791] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 16:06:23 (1775765183) [ 1802.379298] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1804.052804] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 16:06:26 (1775765186) [ 1804.535833] LNet: 93828:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.546952] LNetError: 93828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.580644] LNet: Removed LNI 192.168.204.45@tcp [ 1805.231221] Key type .llcrypt unregistered [ 1805.236451] Key type ._llcrypt unregistered [ 1811.884374] Key type ._llcrypt registered [ 1811.887930] Key type .llcrypt registered [ 1812.165167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1812.193638] alg: No test for adler32 (adler32-zlib) [ 1813.142641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1813.173258] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1821.388645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.145@tcp [ 1828.126888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.145@tcp [ 1829.471075] LNet: 94246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1829.478967] LNetError: 94246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1829.496867] LNet: Removed LNI 192.168.204.45@tcp [ 1830.065443] Key type .llcrypt unregistered [ 1830.078574] Key type ._llcrypt unregistered [ 1842.629438] Key type ._llcrypt registered [ 1842.637787] Key type .llcrypt registered [ 1844.087136] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1846.139741] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1848.312809] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 16:07:10 (1775765230) [ 1848.953117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1848.977238] alg: No test for adler32 (adler32-zlib) [ 1850.016245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.194665] LNet: 95334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.229183] Key type .llcrypt unregistered [ 1851.231622] Key type ._llcrypt unregistered [ 1857.372293] Key type ._llcrypt registered [ 1857.375294] Key type .llcrypt registered [ 1857.648497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1857.683402] alg: No test for adler32 (adler32-zlib) [ 1858.675792] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1858.679853] LNet: 95590:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1858.692952] LNet: 95590:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1858.732869] LNetError: 95602:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.48@tcp is being used as a gateway but routing feature is not turned on [ 1858.750061] LNetError: 95602:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.204.48@tcp has gone from up to down [ 1859.349757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1859.388631] LNet: 95648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.401085] LNetError: 95648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1859.421549] LNet: Removed LNI 192.168.204.45@tcp [ 1859.450265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.562776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1859.597386] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1859.604486] LNet: 95655:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1859.613728] LNet: 95655:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 1859.625387] LNetError: 95655:0:(peer.c:366:lnet_peer_ni_del_locked()) Peer NI 192.168.204.46@tcp is a gateway. Can not delete it [ 1859.632239] LNetError: 95655:0:(peer.c:2166:lnet_del_peer_ni()) 192.168.204.46@tcp is a router. Can not be deleted [ 1859.646504] LNet: 95655:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1859.658179] LNetError: 95652:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.48@tcp is being used as a gateway but routing feature is not turned on [ 1859.693957] LNetError: 95652:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 6 previous similar messages [ 1859.703753] LNetError: 95652:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.204.48@tcp has gone from up to down [ 1859.713038] LNetError: 95652:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1868.078569] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 16:07:30 (1775765250) [ 1870.643087] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg445-server [ 1872.497983] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 16:07:35 (1775765255) [ 1874.638378] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg445-server [ 1876.446443] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 16:07:39 (1775765259) [ 1878.188703] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 1880.175300] Key type lgssc registered [ 1881.508177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1901.211857] Key type lgssc unregistered [ 1901.570953] LNet: 98076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.579121] LNetError: 98076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.594505] LNet: Removed LNI 192.168.204.45@tcp [ 1902.179679] Key type .llcrypt unregistered [ 1902.182812] Key type ._llcrypt unregistered [ 1916.167385] Key type ._llcrypt registered [ 1916.170425] Key type .llcrypt registered [ 1917.362947] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 16:08:20 (1775765300) [ 1917.724677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1917.740154] alg: No test for adler32 (adler32-zlib) [ 1918.631489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.723451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1918.764732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1918.805290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1918.862915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1924.983387] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 16:08:27 (1775765307) [ 1925.268604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1925.290943] LNet: 99483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.318719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1925.380553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1925.405810] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1926.360498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1926.385226] LNet: 99627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1926.389657] LNetError: 99627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1926.404931] LNet: Removed LNI 192.168.204.45@tcp [ 1926.424426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1926.474982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1926.504443] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1927.007104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1927.037325] LNetError: 99655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1927.051444] LNet: Removed LNI 192.168.204.45@tcp [ 1927.092942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1927.155439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1927.201130] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1934.349390] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 16:08:37 (1775765317) [ 1934.557456] LNet: 100246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1934.573270] LNet: 100246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1934.585464] LNetError: 100246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1934.602380] LNet: Removed LNI 192.168.204.45@tcp [ 1935.216759] Key type .llcrypt unregistered [ 1935.221876] Key type ._llcrypt unregistered [ 1941.652904] Key type ._llcrypt registered [ 1941.654707] Key type .llcrypt registered [ 1941.858162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1941.870426] alg: No test for adler32 (adler32-zlib) [ 1942.825834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1942.917577] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1942.921052] LNet: *** cfs_fail_loc=e003, val=100*** [ 1943.644070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1943.659610] LNet: *** cfs_fail_loc=e003, val=100*** [ 1943.662129] LNet: Skipped 2 previous similar messages [ 1943.667221] Ping reply received with mapping: [ 1943.669052] found metadata reply NI[192.168.204.45@tcp] [ 1943.671494] 2d cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1943.674879] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1943.677940] 5a 5a 5a 5a 5a 5a 5a 5a [ 1943.805608] LNet: 100644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.814522] LNetError: 100644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.870334] LNet: Removed LNI 192.168.204.45@tcp [ 1945.540516] Key type .llcrypt unregistered [ 1945.544113] Key type ._llcrypt unregistered [ 1957.261086] Key type ._llcrypt registered [ 1957.263448] Key type .llcrypt registered [ 1958.563253] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 16:09:01 (1775765341) [ 1959.248888] Key type .llcrypt unregistered [ 1959.251594] Key type ._llcrypt unregistered [ 1965.188907] Key type ._llcrypt registered [ 1965.190453] Key type .llcrypt registered [ 1965.408581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1965.444960] alg: No test for adler32 (adler32-zlib) [ 1966.617595] LNetError: 101690:0:(api-ni.c:2731:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1967.070522] Key type .llcrypt unregistered [ 1967.075310] Key type ._llcrypt unregistered [ 1972.517520] Key type ._llcrypt registered [ 1972.520253] Key type .llcrypt registered [ 1973.803625] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 16:09:16 (1775765356) [ 1974.538730] Key type .llcrypt unregistered [ 1974.541447] Key type ._llcrypt unregistered [ 1980.694121] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1980.708695] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1983.204145] Key type ._llcrypt registered [ 1983.206285] Key type .llcrypt registered [ 1983.382613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1983.415493] alg: No test for adler32 (adler32-zlib) [ 1984.323360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1984.396948] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1992.513866] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 16:09:35 (1775765375) [ 1992.701507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.724342] LNet: 103472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.733231] LNetError: 103472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1992.747702] LNet: Removed LNI 10.1.2.3@tcp [ 1992.773814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.397453] Key type .llcrypt unregistered [ 1993.400505] Key type ._llcrypt unregistered [ 1998.428386] Key type ._llcrypt registered [ 1998.433015] Key type .llcrypt registered [ 1998.692460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1998.721062] alg: No test for adler32 (adler32-zlib) [ 1999.659029] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 1999.712612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 1999.890502] LNet: 103777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1999.897868] LNetError: 103777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.937307] LNet: Removed LNI 192.168.204.45@tcp [ 2001.684692] Key type .llcrypt unregistered [ 2001.687236] Key type ._llcrypt unregistered [ 2013.178076] Key type ._llcrypt registered [ 2013.181967] Key type .llcrypt registered [ 2014.916117] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 16:09:57 (1775765397) [ 2015.859328] Key type .llcrypt unregistered [ 2015.861977] Key type ._llcrypt unregistered [ 2021.565173] Key type ._llcrypt registered [ 2021.566653] Key type .llcrypt registered [ 2021.673797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2021.696234] alg: No test for adler32 (adler32-zlib) [ 2029.121461] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 16:10:11 (1775765411) [ 2029.296284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2029.333684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2029.450643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2029.502354] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2030.580509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2030.641169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2030.689480] LNet: Removed LNI 192.168.204.45@tcp [ 2030.917438] LNet: 105522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.257625] Key type .llcrypt unregistered [ 2031.259259] Key type ._llcrypt unregistered [ 2042.436859] Key type ._llcrypt registered [ 2042.445474] Key type .llcrypt registered [ 2044.195788] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 16:10:26 (1775765426) [ 2044.907243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2044.923503] alg: No test for adler32 (adler32-zlib) [ 2045.831767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2045.898467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2045.927108] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2047.103394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 2047.142546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.45@tcp [ 2047.159328] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2047.162785] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2047.166266] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2047, 0, 0 [ 2047.171217] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2047.173490] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2047.177815] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2047.186208] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2047, 0, 0 [ 2047.192825] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2047.196601] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2047.205862] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2047.212551] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2047, 0, 2097 [ 2047.224220] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2047.231529] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2047.245311] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2047.248166] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2047, 0, 2097 [ 2047.259903] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2054.885489] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 16:10:37 (1775765437) [ 2055.549687] LNet: 107062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2055.560235] LNetError: 107062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2055.588862] LNet: Removed LNI 192.168.204.45@tcp [ 2056.296065] Key type .llcrypt unregistered [ 2056.301053] Key type ._llcrypt unregistered [ 2062.868889] Key type ._llcrypt registered [ 2062.871812] Key type .llcrypt registered [ 2063.222240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2063.245882] alg: No test for adler32 (adler32-zlib) [ 2064.219848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2064.260563] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2071.757208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.145@tcp [ 2080.175869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2080.229316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2080.245632] LNet: Added LNI 192.168.204.45@tcp1 [8/256/0/180] [ 2081.121403] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2081.336596] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2081.350245] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2081.377717] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2083.717780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2083.739602] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2093.700229] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 16:11:16 (1775765476) [ 2094.094833] LNet: 108570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2094.111545] LNetError: 108570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2094.120745] LNet: Removed LNI 192.168.204.45@tcp [ 2095.135275] LNet: Removed LNI 192.168.204.45@tcp1 [ 2095.140277] LNetError: 108570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2095.148254] LNetError: 108570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2095.180636] LNet: Removed LNI 10.1.2.3@tcp99 [ 2095.920949] Key type .llcrypt unregistered [ 2095.924599] Key type ._llcrypt unregistered [ 2102.336264] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2102.345954] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2104.952424] Key type ._llcrypt registered [ 2104.954488] Key type .llcrypt registered [ 2105.133206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2105.168072] alg: No test for adler32 (adler32-zlib) [ 2106.156871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.220736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2106.255305] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2106.965573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2106.992612] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2109.443828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2109.490812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2109.545591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2109.566159] LNetError: 109314:0:(peer.c:2174:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2109.593031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2109.636550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2110.183342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2110.286705] LNet: 109334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.300333] LNetError: 109334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2111.327254] LNet: Removed LNI 192.168.204.45@tcp [ 2111.330223] LNetError: 109334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2111.346059] LNet: Removed LNI 10.1.2.3@tcp [ 2111.878840] Key type .llcrypt unregistered [ 2111.886600] Key type ._llcrypt unregistered [ 2121.316476] Key type ._llcrypt registered [ 2121.318850] Key type .llcrypt registered [ 2122.472224] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 16:11:45 (1775765505) [ 2123.001634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2123.013885] alg: No test for adler32 (adler32-zlib) [ 2123.877669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.958441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2123.984874] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2124.598831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.45@tcp [ 2124.719202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg445-client@tcp [ 2130.632283] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 16:11:53 (1775765513) [ 2130.856788] LNet: 110867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2130.867420] LNetError: 110867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2130.877726] LNet: Removed LNI 192.168.204.45@tcp [ 2131.463904] Key type .llcrypt unregistered [ 2131.467968] Key type ._llcrypt unregistered [ 2136.522222] Key type ._llcrypt registered [ 2136.525041] Key type .llcrypt registered [ 2136.708685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2136.734605] alg: No test for adler32 (adler32-zlib) [ 2137.597786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2137.633042] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2137.659794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2143.328526] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 16:12:06 (1775765526) [ 2144.717385] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2146.042793] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 16:12:09 (1775765529) [ 2147.527447] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2149.269922] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 16:12:12 (1775765532) [ 2149.813200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2149.864041] LNet: 112286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.876799] LNetError: 112286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2149.905518] LNet: Removed LNI 192.168.204.45@tcp [ 2149.929340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.994290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2150.050646] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2150.787328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2150.836978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2158.139381] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 16:12:20 (1775765540) [ 2158.366117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2158.384823] LNet: 112995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.394103] LNetError: 112995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2158.406475] LNet: Removed LNI 192.168.204.45@tcp [ 2158.422128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.467894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2158.494612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2159.093759] Key type .llcrypt unregistered [ 2159.095994] Key type ._llcrypt unregistered [ 2170.820044] Key type ._llcrypt registered [ 2170.821862] Key type .llcrypt registered [ 2172.583645] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 16:12:34 (1775765554) [ 2173.011025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2173.034107] alg: No test for adler32 (adler32-zlib) [ 2174.072877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2174.128420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2174.165407] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2174.957306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2175.049916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2182.362585] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 16:12:45 (1775765565) [ 2182.456797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2182.488243] LNet: 114544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.502418] LNetError: 114544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2182.523880] LNet: Removed LNI 192.168.204.45@tcp [ 2182.555721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.651742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2182.700455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2189.669675] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 16:12:52 (1775765572) [ 2189.930850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.994807] LNet: 115131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2190.031205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2190.097078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2190.715349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2190.764627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2190.865682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2191.345932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2191.392425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2191.457895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2191.475731] LNet: 115164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.493949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2191.556153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2192.086624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2192.158959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2192.230469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2192.917716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2192.981238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2199.436741] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 16:13:02 (1775765582) [ 2199.595406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2199.621304] LNet: 115759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2199.663609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2199.741097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2199.778538] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2200.549495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2200.577153] LNet: Added LNI 192.168.204.45@tcp2 [8/256/0/180] [ 2201.369259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2201.407123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2201.630686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2201.670419] LNetError: 116043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2201.688870] LNet: Removed LNI 192.168.204.45@tcp [ 2201.704158] LNet: Removed LNI 192.168.204.45@tcp2 [ 2201.729262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.799981] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2201.806412] LNet: Added LNI 192.168.204.45@tcp2 [8/256/0/180] [ 2201.811760] LNet: 116049:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2202.330411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2202.553644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2203.615237] LNet: Removed LNI 192.168.204.45@tcp [ 2203.617259] LNetError: 116087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2203.639059] LNetError: 116087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2203.668499] LNet: Removed LNI 192.168.204.45@tcp2 [ 2203.703943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.769990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2203.792498] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2204.626543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2204.646428] LNet: Added LNI 192.168.204.45@tcp2 [8/256/0/180] [ 2205.646673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2205.691712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2205.727896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2205.869532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2205.886825] LNet: 116371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2205.892074] LNet: 116371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2205.899166] LNetError: 116371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2205.903794] LNet: Removed LNI 192.168.204.45@tcp [ 2206.962102] LNet: Removed LNI 192.168.204.45@tcp2 [ 2207.002587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2207.081387] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2207.094784] LNet: Added LNI 192.168.204.45@tcp2 [8/256/0/180] [ 2207.103800] LNet: 116377:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2207.694804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2207.979199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2207.997602] LNet: Removed LNI 192.168.204.45@tcp [ 2209.062241] LNet: Removed LNI 192.168.204.45@tcp2 [ 2209.102583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.197093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2209.233809] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2210.097532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2210.122488] LNet: Added LNI 192.168.204.45@tcp2 [8/256/0/180] [ 2211.185574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2211.229963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2211.464273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2211.485427] LNetError: 116694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2211.494241] LNetError: 116694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2211.505713] LNet: Removed LNI 192.168.204.45@tcp [ 2211.520262] LNet: Removed LNI 192.168.204.45@tcp2 [ 2211.552731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2211.630931] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2211.641512] LNet: Added LNI 192.168.204.45@tcp2 [8/256/0/180] [ 2211.651478] LNet: 116700:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2212.419794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2212.690930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2213.727863] LNet: Removed LNI 192.168.204.45@tcp [ 2213.736791] LNet: Removed LNI 192.168.204.45@tcp2 [ 2213.756239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2213.806288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2213.830938] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2214.636262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2214.649503] LNet: Added LNI 192.168.204.45@tcp2 [8/256/0/180] [ 2215.947677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2216.016842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2216.115572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2216.451360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2216.475382] LNet: 117022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2216.481451] LNet: 117022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2217.503132] LNet: Removed LNI 192.168.204.45@tcp [ 2218.533836] LNet: Removed LNI 192.168.204.45@tcp2 [ 2218.605530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.725444] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2218.730548] LNet: Added LNI 192.168.204.45@tcp2 [8/256/0/180] [ 2218.735601] LNet: 117028:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2219.593483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2219.908318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2219.964875] LNetError: 117065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2219.972905] LNetError: 117065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2219.983622] LNet: Removed LNI 192.168.204.45@tcp [ 2221.029896] LNet: Removed LNI 192.168.204.45@tcp2 [ 2221.063282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2221.149349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2221.172748] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2221.726124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2221.753298] LNet: Added LNI 192.168.204.45@tcp2 [8/256/0/180] [ 2222.722295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2222.784859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2222.892444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2223.224600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2224.288288] LNet: Removed LNI 192.168.204.45@tcp [ 2224.304655] LNet: Removed LNI 192.168.204.45@tcp2 [ 2224.341577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2224.429256] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2224.442379] LNet: Added LNI 192.168.204.45@tcp2 [8/256/0/180] [ 2224.446236] LNet: 117368:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2225.271882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2225.561994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.656093] LNet: Removed LNI 192.168.204.45@tcp [ 2226.668704] LNet: Removed LNI 192.168.204.45@tcp2 [ 2226.758503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2226.857312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2226.878962] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2227.409718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2227.455841] LNet: Added LNI 192.168.204.45@tcp2 [8/256/0/180] [ 2228.741771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2228.782828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2228.814622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2228.887356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2229.473195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2230.559161] LNet: Removed LNI 192.168.204.45@tcp [ 2231.592589] LNet: Removed LNI 192.168.204.45@tcp2 [ 2231.607638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2231.653825] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2231.658550] LNet: Added LNI 192.168.204.45@tcp2 [8/256/0/180] [ 2232.023439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2240.399898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2242.352759] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 16:13:44 (1775765624) [ 2242.691571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2242.758358] LNet: 118487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.764742] LNet: 118487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2242.776708] LNetError: 118487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2242.787775] LNetError: 118487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2242.799411] LNet: Removed LNI 192.168.204.45@tcp [ 2243.815370] LNet: Removed LNI 192.168.204.45@tcp2 [ 2243.844519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.079364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2244.112835] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2244.909725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2244.939099] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2245.993159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2246.038502] LNet: Removed LNI 192.168.204.45@tcp [ 2246.051741] LNet: Removed LNI 10.1.2.3@tcp [ 2246.069339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2246.135845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2246.157930] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2246.162811] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2246.651520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2246.708691] LNet: Removed LNI 192.168.204.45@tcp [ 2246.736803] LNet: Removed LNI 10.1.2.3@tcp [ 2253.275036] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 16:13:56 (1775765636) [ 2253.494559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2253.553871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2253.638450] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2253.770230] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2253.779717] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2253.786751] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2256.167728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2256.191737] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2256.974496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2257.005846] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2259.123189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.45@tcp [ 2259.162529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2259.227789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.45@tcp -m PUT -e local_timeout -r 1 [ 2259.296269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2259.374486] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2259.378394] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2259.562282] LNetError: 119429:0:(lib-move.c:4969:lnet_parse()) 192.168.204.45@tcp, src 192.168.204.45@tcp: Dropping PUT (error -108 looking up sender) [ 2261.471242] LNet: Removed LNI 192.168.204.45@tcp [ 2261.488957] LNet: Removed LNI 10.1.2.3@tcp [ 2261.904384] Key type .llcrypt unregistered [ 2261.907989] Key type ._llcrypt unregistered [ 2272.830653] Key type ._llcrypt registered [ 2272.833465] Key type .llcrypt registered [ 2273.891185] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 16:14:16 (1775765656) [ 2274.480514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2274.501287] alg: No test for adler32 (adler32-zlib) [ 2275.436793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2275.786585] LNet: 120740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.399428] Key type .llcrypt unregistered [ 2276.403076] Key type ._llcrypt unregistered [ 2281.934473] Key type ._llcrypt registered [ 2281.938082] Key type .llcrypt registered [ 2282.125417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2282.143405] alg: No test for adler32 (adler32-zlib) [ 2283.030756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2283.054288] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2290.403506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.145@tcp [ 2290.605839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2291.257571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.145@tcp [ 2291.608521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2291.658244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2291.695927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2291.731672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2291.767942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2291.805298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2291.845471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2291.888534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2291.922955] LNet: 121189:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2291.946609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2291.984573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2292.023949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2292.065976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2292.105331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2292.135980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2292.756678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2292.969848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2293.001314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2293.027744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2293.055857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2293.079350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2293.100957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2293.122988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2293.147613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2293.161664] LNet: 121306:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2293.191510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2293.221571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2293.254635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2293.280394] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.145@tcp [ 2293.291597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2293.323958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2293.359967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2293.878257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.145@tcp [ 2294.110290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2294.143572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2294.182107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2294.211775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2294.238461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2294.268444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2294.296921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2294.326736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2294.343969] LNet: 121423:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2294.366525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2294.403710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2294.435793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2294.466824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2294.502962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2294.541949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2295.101216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2295.297963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2295.322325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2295.345472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2295.370223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2295.392844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2295.414600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2295.444164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2295.488118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2295.533134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2295.567320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2295.602240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2295.630791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2295.670836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2295.701737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2295.944614] LNet: 121587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2295.952607] LNetError: 121587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2295.974951] LNet: Removed LNI 192.168.204.45@tcp [ 2296.384985] Key type .llcrypt unregistered [ 2296.392446] Key type ._llcrypt unregistered [ 2308.108676] Key type ._llcrypt registered [ 2308.110929] Key type .llcrypt registered [ 2309.295467] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 16:14:52 (1775765692) [ 2309.675102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2309.695438] alg: No test for adler32 (adler32-zlib) [ 2310.601811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.917917] LNet: 122473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.451378] Key type .llcrypt unregistered [ 2311.454271] Key type ._llcrypt unregistered [ 2317.162570] Key type ._llcrypt registered [ 2317.165216] Key type .llcrypt registered [ 2317.389749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2317.440537] alg: No test for adler32 (adler32-zlib) [ 2318.377883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2318.411130] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2325.624323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.145@tcp [ 2334.477715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.520367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2334.545682] LNet: Added LNI 192.168.204.45@tcp1 [8/256/0/180] [ 2336.043868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2336.863867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.145@tcp [ 2337.276351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2337.332231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2337.402548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2337.462689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2337.526577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2337.601577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2337.667328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2337.752187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2337.798061] LNet: 123125:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2337.828229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2337.859253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2337.891260] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.145@tcp [ 2337.924441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2337.991331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2338.047709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2338.088425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2338.961544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2339.372434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2339.432616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2339.469136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2339.532653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2339.579340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2339.640071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2339.734678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2339.800815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2339.847331] LNet: 123242:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2339.882077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2339.948401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2339.986047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2340.044236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2340.112581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2340.185208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2340.922127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.145@tcp [ 2341.147359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2341.186546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2341.220858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2341.314216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2341.403816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2341.445398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2341.517850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2341.572928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2341.607364] LNet: 123359:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2341.640154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2341.689928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2341.739065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2341.804349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2341.842159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2341.910909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2342.778794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2343.031170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2343.077029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2343.136502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2343.205772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2343.241430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2343.284655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2343.326585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2343.365060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2343.409451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2343.445756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2343.491336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2343.542671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2343.618435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2343.668543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2349.448836] LNet: 123558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2349.463285] LNetError: 123558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2349.486351] LNet: Removed LNI 192.168.204.45@tcp [ 2349.497735] LNet: Removed LNI 192.168.204.45@tcp1 [ 2350.159857] Key type .llcrypt unregistered [ 2350.170931] Key type ._llcrypt unregistered [ 2366.553669] Key type ._llcrypt registered [ 2366.557183] Key type .llcrypt registered [ 2367.854548] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 16:15:50 (1775765750) [ 2375.294818] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 16:15:58 (1775765758) [ 2381.535169] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 16:16:04 (1775765764) [ 2388.413261] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 16:16:11 (1775765771) [ 2395.761708] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 16:16:18 (1775765778) [ 2404.029966] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 16:16:26 (1775765786) [ 2411.922588] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 16:16:34 (1775765794) [ 2420.949960] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 16:16:43 (1775765803) [ 2429.464309] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 16:16:52 (1775765812) [ 2436.499853] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 16:16:59 (1775765819) [ 2443.444772] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 16:17:06 (1775765826) [ 2451.571591] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 16:17:14 (1775765834) [ 2459.200898] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 16:17:21 (1775765841) [ 2467.266277] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 16:17:29 (1775765849) [ 2475.280497] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 16:17:38 (1775765858) [ 2483.125745] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 16:17:45 (1775765865) [ 2491.043231] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 16:17:53 (1775765873) [ 2498.688415] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 16:18:01 (1775765881) [ 2499.192990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2499.207130] alg: No test for adler32 (adler32-zlib) [ 2500.174664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2500.325382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2500.350361] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2501.680293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2501.716190] LNet: 134268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2501.727219] LNetError: 134268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2501.740876] LNet: Removed LNI 192.168.204.45@tcp [ 2501.762691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2501.832368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2501.859132] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2509.078278] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 16:18:12 (1775765892) [ 2516.554797] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 16:18:19 (1775765899) [ 2526.115200] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 16:18:28 (1775765908) [ 2534.934368] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 16:18:37 (1775765917) [ 2542.293648] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 16:18:45 (1775765925) [ 2551.177726] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 16:18:53 (1775765933) [ 2559.263962] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 16:19:01 (1775765941) [ 2569.200330] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 16:19:11 (1775765951) [ 2577.037894] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 16:19:19 (1775765959) [ 2586.610824] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 16:19:29 (1775765969) [ 2595.077890] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 16:19:37 (1775765977) [ 2602.378522] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 16:19:45 (1775765985) [ 2610.254568] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2391 sec ========== 16:19:52 (1775765992) [ 2610.634540] LNet: 141739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2610.639434] LNetError: 141739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2610.659099] LNet: Removed LNI 192.168.204.45@tcp [ 2611.479463] Key type .llcrypt unregistered [ 2611.481599] Key type ._llcrypt unregistered [ 2631.670065] Key type ._llcrypt registered [ 2631.673681] Key type .llcrypt registered [ 2632.632802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2632.681131] alg: No test for adler32 (adler32-zlib) [ 2633.916694] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 2634.195193] LNet: Added LNI 192.168.204.45@tcp [8/256/0/180] [ 2636.066241] Key type lgssc registered [ 2637.095277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2717.840091] Lustre: Mounted lustre-client [ 2723.338808] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2738.072907] LustreError: 144196:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff96070aa89800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2738.186570] Lustre: Unmounted lustre-client [ 2772.390736] Key type lgssc unregistered [ 2772.637416] LNet: 144678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2772.646463] LNetError: 144678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2772.673309] LNet: Removed LNI 192.168.204.45@tcp [ 2773.447642] Key type .llcrypt unregistered [ 2773.449369] Key type ._llcrypt unregistered