[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 447863792 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003368] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.006004] kvm-guest: setup PV IPIs [ 0.008923] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011136] LSM: Security Framework initializing [ 0.012049] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014087] *** VALIDATE selinux *** [ 0.022740] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027022] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029097] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030109] *** VALIDATE tmpfs *** [ 0.032154] *** VALIDATE proc *** [ 0.033270] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035304] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042538] debug: unmapping init [mem 0xffffffffaf659000-0xffffffffaf660fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045684] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.047959] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052296] rcu: Hierarchical SRCU implementation. [ 0.054512] smp: Bringing up secondary CPUs ... [ 0.055515] x86: Booting SMP configuration: [ 0.056023] .... node #0, CPUs: #1 #2 #3 [ 0.059452] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139431] node 0 deferred pages initialised in 76ms [ 0.143206] devtmpfs: initialized [ 0.144082] x86/mm: Memory block size: 128MB [ 0.147778] gcov: version magic: 0x41383552 [ 0.150170] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154124] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157335] pinctrl core: initialized pinctrl subsystem [ 0.159190] [ 0.159682] ************************************************************* [ 0.162011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164010] ** ** [ 0.166014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168012] ** ** [ 0.170011] ** This means that this kernel is built to expose internal ** [ 0.172013] ** IOMMU data structures, which may compromise security on ** [ 0.174013] ** your system. ** [ 0.176013] ** ** [ 0.178013] ** If you see this message and you are not debugging the ** [ 0.181016] ** kernel, report this immediately to your vendor! ** [ 0.183012] ** ** [ 0.185015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187014] ************************************************************* [ 0.189653] NET: Registered protocol family 16 [ 0.191401] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.194055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.197063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.200124] cpuidle: using governor menu [ 0.201735] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204522] PCI: Using configuration type 1 for base access [ 0.207150] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217136] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.222060] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228230] cryptd: max_cpu_qlen set to 1000 [ 0.233590] ACPI: Added _OSI(Module Device) [ 0.234020] ACPI: Added _OSI(Processor Device) [ 0.235020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.237019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242258] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.250444] ACPI: Interpreter enabled [ 0.253081] ACPI: PM: (supports S0 S3 S4 S5) [ 0.255011] ACPI: Using IOAPIC for interrupt routing [ 0.256124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.259432] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.268000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273035] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277121] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284558] acpiphp: Slot [2] registered [ 0.286145] acpiphp: Slot [3] registered [ 0.287136] acpiphp: Slot [4] registered [ 0.289145] acpiphp: Slot [5] registered [ 0.291161] acpiphp: Slot [6] registered [ 0.292151] acpiphp: Slot [7] registered [ 0.293170] acpiphp: Slot [8] registered [ 0.295120] acpiphp: Slot [9] registered [ 0.297130] acpiphp: Slot [10] registered [ 0.298329] acpiphp: Slot [11] registered [ 0.300096] acpiphp: Slot [12] registered [ 0.301100] acpiphp: Slot [13] registered [ 0.303108] acpiphp: Slot [14] registered [ 0.304146] acpiphp: Slot [15] registered [ 0.306103] acpiphp: Slot [16] registered [ 0.307091] acpiphp: Slot [17] registered [ 0.309081] acpiphp: Slot [18] registered [ 0.310105] acpiphp: Slot [19] registered [ 0.312124] acpiphp: Slot [20] registered [ 0.313171] acpiphp: Slot [21] registered [ 0.315105] acpiphp: Slot [22] registered [ 0.316135] acpiphp: Slot [23] registered [ 0.317104] acpiphp: Slot [24] registered [ 0.319102] acpiphp: Slot [25] registered [ 0.320126] acpiphp: Slot [26] registered [ 0.321113] acpiphp: Slot [27] registered [ 0.323109] acpiphp: Slot [28] registered [ 0.324086] acpiphp: Slot [29] registered [ 0.325081] acpiphp: Slot [30] registered [ 0.326108] acpiphp: Slot [31] registered [ 0.328095] PCI host bridge to bus 0000:00 [ 0.329021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.334026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.336034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.338022] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.340028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.342197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.346268] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.350561] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.357432] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.362056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.364023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.367019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.368017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.370566] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.373858] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.376069] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.379759] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.384015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.394022] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.398022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.403917] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.411025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.415021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.428028] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.436184] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.442017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.448017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.461016] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.471030] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.473377] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.475480] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.478465] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.481322] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.487180] iommu: Default domain type: Passthrough [ 0.488516] SCSI subsystem initialized [ 0.489203] ACPI: bus type USB registered [ 0.491159] usbcore: registered new interface driver usbfs [ 0.493151] usbcore: registered new interface driver hub [ 0.495123] usbcore: registered new device driver usb [ 0.497221] pps_core: LinuxPPS API ver. 1 registered [ 0.499013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.504069] PTP clock support registered [ 0.506158] EDAC MC: Ver: 3.0.0 [ 0.508165] PCI: Using ACPI for IRQ routing [ 0.509972] NetLabel: Initializing [ 0.510014] NetLabel: domain hash size = 128 [ 0.511012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.512056] NetLabel: unlabeled traffic allowed by default [ 0.514135] vgaarb: loaded [ 0.516372] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.517016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.527668] clocksource: Switched to clocksource kvm-clock [ 0.646258] VFS: Disk quotas dquot_6.6.0 [ 0.648273] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.650803] *** VALIDATE ramfs *** [ 0.651858] *** VALIDATE hugetlbfs *** [ 0.653561] pnp: PnP ACPI init [ 0.656237] pnp: PnP ACPI: found 6 devices [ 0.673925] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.677266] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.679289] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.681710] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.684256] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.686929] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.689718] NET: Registered protocol family 2 [ 0.692490] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.697543] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.701692] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.706586] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.710254] TCP: Hash tables configured (established 65536 bind 65536) [ 0.714166] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.717137] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.719936] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.723071] NET: Registered protocol family 1 [ 0.726220] RPC: Registered named UNIX socket transport module. [ 0.727973] RPC: Registered udp transport module. [ 0.729269] RPC: Registered tcp transport module. [ 0.733158] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.736570] NET: Registered protocol family 44 [ 0.737848] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.739564] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.741508] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.743727] PCI: CLS 0 bytes, default 64 [ 0.745224] Unpacking initramfs... [ 2.138133] debug: unmapping init [mem 0xffff913cbcc64000-0xffff913cbffcffff] [ 2.141694] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.143983] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.146817] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.639578] Initialise system trusted keyrings [ 2.641369] Key type blacklist registered [ 2.643354] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.653555] zbud: loaded [ 2.656894] *** VALIDATE nfs *** [ 2.658217] *** VALIDATE nfs4 *** [ 2.659768] pstore: using deflate compression [ 2.662961] Platform Keyring initialized [ 2.801979] NET: Registered protocol family 38 [ 2.803467] Key type asymmetric registered [ 2.805363] Asymmetric key parser 'x509' registered [ 2.807734] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.814933] io scheduler mq-deadline registered [ 2.818628] io scheduler kyber registered [ 2.821329] io scheduler bfq registered [ 2.823721] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.828273] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.831753] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.834041] ACPI: Power Button [PWRF] [ 2.936062] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.044063] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.153790] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.185804] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.222959] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.228933] Non-volatile memory driver v1.3 [ 3.230699] Linux agpgart interface v0.103 [ 3.270780] virtio_blk virtio1: [vda] 133832 512-byte logical blocks (68.5 MB/65.3 MiB) [ 3.273439] vda: detected capacity change from 0 to 68521984 [ 3.289493] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.293301] vdb: detected capacity change from 0 to 1073741824 [ 3.301296] libphy: Fixed MDIO Bus: probed [ 3.308448] usbcore: registered new interface driver usbserial_generic [ 3.312505] usbserial: USB Serial support registered for generic [ 3.314564] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.318502] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.320532] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.322921] mousedev: PS/2 mouse device common for all mice [ 3.325709] rtc_cmos 00:05: RTC can wake from S4 [ 3.329354] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.334512] rtc_cmos 00:05: registered as rtc0 [ 3.336441] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.341082] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.343516] intel_pstate: CPU model not supported [ 3.360707] hid: raw HID events driver (C) Jiri Kosina [ 3.363912] usbcore: registered new interface driver usbhid [ 3.366020] usbhid: USB HID core driver [ 3.367353] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.372047] drop_monitor: Initializing network drop monitor service [ 3.379317] Initializing XFRM netlink socket [ 3.381464] NET: Registered protocol family 10 [ 3.385826] Segment Routing with IPv6 [ 3.387088] NET: Registered protocol family 17 [ 3.388778] mpls_gso: MPLS GSO support [ 3.396588] RAS: Correctable Errors collector initialized. [ 3.399928] AVX version of gcm_enc/dec engaged. [ 3.401595] AES CTR mode by8 optimization enabled [ 3.500512] sched_clock: Marking stable (3500487128, 0)->(4415585069, -915097941) [ 3.504296] registered taskstats version 1 [ 3.506466] Loading compiled-in X.509 certificates [ 3.508536] zswap: loaded using pool lzo/zbud [ 3.544206] Key type big_key registered [ 3.559069] Key type encrypted registered [ 3.560598] ima: No TPM chip found, activating TPM-bypass! [ 3.562758] ima: Allocated hash algorithm: sha1 [ 3.564337] ima: No architecture policies found [ 3.566106] evm: Initialising EVM extended attributes: [ 3.568073] evm: security.selinux [ 3.569434] evm: security.ima [ 3.570698] evm: security.capability [ 3.572299] evm: HMAC attrs: 0x1 [ 3.575218] rtc_cmos 00:05: setting system clock to 2025-11-16 22:47:32 UTC (1763333252) [ 3.583369] debug: unmapping init [mem 0xffffffffb0603000-0xffffffffb07fffff] [ 3.586435] debug: unmapping init [mem 0xffffffffaf382000-0xffffffffaf658fff] [ 3.592075] Write protecting the kernel read-only data: 28672k [ 3.595510] debug: unmapping init [mem 0xffffffffada03000-0xffffffffadbfffff] [ 3.598295] debug: unmapping init [mem 0xffffffffae314000-0xffffffffae3fffff] [ 3.638737] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.647752] systemd[1]: Detected virtualization kvm. [ 3.649769] systemd[1]: Detected architecture x86-64. [ 3.651758] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.682287] systemd[1]: No hostname configured. [ 3.684138] systemd[1]: Set hostname to . [ 3.686903] random: systemd: uninitialized urandom read (16 bytes read) [ 3.690775] systemd[1]: Initializing machine ID from random generator. [ 3.766665] random: ln: uninitialized urandom read (6 bytes read) [ 3.888125] random: systemd: uninitialized urandom read (16 bytes read) [ 3.891379] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.901408] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.907674] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook.[ 4.500919] random: fast init done Starting dracut pre-udev hook... [ 4.568174] device-mapper: uevent: version 1.0.3 [ 4.570995] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.280496] virtio_net virtio0 ens2: renamed from eth0 [ 5.369670] scsi host0: ata_piix [ 5.398391] scsi host1: ata_piix [ 5.400168] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.402932] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.890025] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.078902] random: crng init done [ 10.080282] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.579884] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.802645] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.062362] SELinux: Disabled at runtime. [ 12.122870] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.132057] systemd[1]: Detected virtualization kvm. [ 12.134107] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.690633] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.693950] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.698001] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.702236] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.706575] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.714769] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.723932] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ 12.840158] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.226734] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.581748] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.661044] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.764374] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.788266] EDAC sbridge: Ver: 1.1.2 [ 15.104105] Key type dns_resolver registered [ 15.413319] NFS: Registering the id_resolver key type [ 15.415365] Key type id_resolver registered [ 15.417040] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg413-client login: [ 52.817794] libcfs: loading out-of-tree module taints kernel. [ 52.880473] Key type ._llcrypt registered [ 52.882232] Key type .llcrypt registered [ 53.656081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 53.672362] alg: No test for adler32 (adler32-zlib) [ 55.077643] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 55.638811] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 57.335208] Key type lgssc registered [ 59.313273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.992114] hrtimer: interrupt took 4106748 ns [ 156.328082] Lustre: Mounted lustre-client [ 160.325187] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 181.576068] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing check_logdir /tmp/testlogs/ [ 181.729246] Lustre: lustre-OST0000-osc-ffff913d0316a800: disconnect after 23s idle [ 186.049725] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing yml_node [ 190.484651] Lustre: DEBUG MARKER: Client: 2.16.61.52 [ 192.687994] Lustre: DEBUG MARKER: MDS: 2.16.61.52 [ 194.976539] Lustre: DEBUG MARKER: OSS: 2.16.61.52 [ 196.368027] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Nov 16 17:50:44 EST 2025 [ 209.486429] LustreError: 7951:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff913d0316a800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 209.503580] LustreError: 7951:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 209.530316] Lustre: Unmounted lustre-client [ 222.974911] Key type lgssc unregistered [ 223.181052] LNet: 8396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 223.185568] LNetError: 8396:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 223.198874] LNet: Removed LNI 192.168.204.13@tcp [ 223.677178] Key type .llcrypt unregistered [ 223.683808] Key type ._llcrypt unregistered [ 240.956150] Key type ._llcrypt registered [ 240.966386] Key type .llcrypt registered [ 241.148865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 241.171903] alg: No test for adler32 (adler32-zlib) [ 242.085480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 242.153764] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 242.173574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 245.491670] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 245.716461] LNet: 9268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 245.725906] LNetError: 9268:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 245.749572] LNet: Removed LNI 192.168.204.13@tcp [ 246.371595] Key type .llcrypt unregistered [ 246.373371] Key type ._llcrypt unregistered [ 251.326573] Key type ._llcrypt registered [ 251.328385] Key type .llcrypt registered [ 252.442542] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 17:51:40 (1763333500) [ 252.785286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 252.801160] alg: No test for adler32 (adler32-zlib) [ 253.707830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 253.769708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 253.784065] LNet: 9674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 253.801327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 253.836359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 253.859631] LNet: 9681:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 259.543915] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 17:51:47 (1763333507) [ 259.791730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 259.821258] LNet: 10258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 259.840532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 259.900450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 264.741817] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 17:51:52 (1763333512) [ 264.877696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 264.914134] LNet: 10843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 264.937536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.001453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 269.933585] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 17:51:57 (1763333517) [ 270.051929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 270.079682] LNet: 11421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 270.128961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 270.233336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 275.781485] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 17:52:03 (1763333523) [ 275.885174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 275.926940] LNet: 12006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 275.979726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.083587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 276.389142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 282.352521] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 17:52:09 (1763333529) [ 282.483476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.533251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.655257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 283.085059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 283.615482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 289.869021] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 17:52:17 (1763333537) [ 290.003651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.029112] LNet: 13193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.036383] LNet: 13193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 290.061179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.145902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 296.073917] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 17:52:23 (1763333543) [ 296.222976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.272548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.343355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 296.375563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 296.626858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 296.655270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 296.911494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 296.939704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 297.359487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 297.387343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 297.884723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 297.917278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 298.382723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 298.417435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 298.768900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 298.814373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 299.135694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 299.163409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 299.604472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 299.662118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 300.016360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 300.075933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 300.573947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 300.638053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 301.108377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 301.160887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 301.626740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 301.656087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 307.843442] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 17:52:35 (1763333555) [ 308.100339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.150490] LNet: 14463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 308.159042] LNet: 14463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 308.222804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.308093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 308.336631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 313.819946] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 17:52:41 (1763333561) [ 313.987866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.031450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.098868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 314.122798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 319.471785] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 17:52:47 (1763333567) [ 319.616581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.670307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.725534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 319.755886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 325.243944] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 17:52:53 (1763333573) [ 325.330679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.372261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.415994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 325.445936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 331.032130] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 17:52:58 (1763333578) [ 331.221744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.286568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.367890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 331.430200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 337.345674] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 17:53:05 (1763333585) [ 337.463278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.507815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.626754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 337.655913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 344.152537] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 17:53:11 (1763333591) [ 344.352723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.404530] LNet: 17983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.415601] LNet: 17983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 344.452092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.555839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 344.987599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 349.857356] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 17:53:17 (1763333597) [ 349.924264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.956180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.129331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 350.301129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 355.717298] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 17:53:23 (1763333603) [ 355.948783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.988788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.102384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 356.310933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 362.191770] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 17:53:29 (1763333609) [ 362.410552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.448859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.855216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 363.100325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 368.341855] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 17:53:36 (1763333616) [ 368.410059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.440819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.505711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 368.556030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 373.364311] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 17:53:41 (1763333621) [ 373.536392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.571260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.622324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 373.653560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 379.077568] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 17:53:46 (1763333626) [ 379.219368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.292339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.384704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 385.464264] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 17:53:52 (1763333632) [ 385.751204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.807915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.942937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 392.249423] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 17:53:59 (1763333639) [ 392.532048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.578799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.677043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 393.222538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 398.978898] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 17:54:06 (1763333646) [ 399.069283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.161543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.224977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 399.243807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 405.743446] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 17:54:13 (1763333653) [ 405.949952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.033986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.107313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 406.147499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 411.873281] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 17:54:19 (1763333659) [ 411.966117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.985483] LNet: 24589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.988077] LNet: 24589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 412.010157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.098037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 412.155720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 418.212907] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 17:54:25 (1763333665) [ 418.257859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.291201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.365967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 418.393756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 423.520557] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 17:54:31 (1763333671) [ 423.656935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.696423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.793564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 423.853335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 423.877471] LNetError: 25779:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 423.922844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 423.956217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 430.344102] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 17:54:37 (1763333677) [ 430.424374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.484135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.621349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LEsK3BKSDo [ 430.694782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LEsK3BKSDo [ 430.758927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LEsK3BKSDo [ 436.396258] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 17:54:44 (1763333684) [ 436.565067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.602908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.638539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 436.692926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 436.735859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 441.829966] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 17:54:49 (1763333689) [ 442.045815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.129797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.298045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 444.903937] LNet: Message forwarding will be enabled in 2s [ 447.967202] LNet: Message forwarding enabled [ 448.382856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 448.407818] LNet: Message forwarding will stop in 2s to 6s [ 452.872864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 453.255655] LNet: Message forwarding disabled [ 454.591929] LNet: Message forwarding will be enabled in 2s [ 458.399321] LNet: Message forwarding enabled [ 460.696974] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 17:55:08 (1763333708) [ 460.915769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.335150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.410971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 468.365502] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 17:55:16 (1763333716) [ 468.463505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.532122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.575966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 468.597597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 468.621236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 468.646231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 468.672876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 468.699621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 468.723134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 468.745976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 468.771773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 468.802635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 468.840544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 468.892277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 468.930356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 468.965532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 469.013798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 469.051824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 469.102756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 469.135923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 469.173830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 469.190831] LNetError: 28849:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 474.067955] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 17:55:21 (1763333721) [ 474.270108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.444748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.510355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 480.791159] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 17:55:28 (1763333728) [ 482.653420] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg413-server [ 484.337339] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 17:55:32 (1763333732) [ 484.667830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.719414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.799385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 484.817357] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 486.445516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[14-18]@tcp [ 486.503210] LNet: 30565:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 486.588415] LNetError: 30354:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.15@tcp is being used as a gateway but routing feature is not turned on [ 486.605647] LNetError: 30354:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.15@tcp has gone from up to down [ 487.588512] LNetError: 30354:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.14@tcp is being used as a gateway but routing feature is not turned on [ 487.595734] LNetError: 30354:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 488.612328] LNetError: 30354:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.14@tcp is being used as a gateway but routing feature is not turned on [ 488.648323] LNetError: 30354:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 490.678624] LNetError: 30354:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.14@tcp is being used as a gateway but routing feature is not turned on [ 490.689726] LNetError: 30354:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 493.433476] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 17:55:40 (1763333740) [ 494.763277] LNetError: 30354:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.14@tcp is being used as a gateway but routing feature is not turned on [ 494.779604] LNetError: 30354:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 495.239533] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg413-server [ 496.898984] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 17:55:44 (1763333744) [ 496.968734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.033184] LNetError: 31493:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 497.059090] LNet: Removed LNI 192.168.204.13@tcp [ 497.086949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.167460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 497.203166] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 497.981921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[14-19/2]@tcp [ 497.998847] LNet: 31617:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 498.003865] LNet: 31617:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 498.032090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[14-19/2]@tcp [ 498.037118] LNetError: 31503:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.14@tcp has gone from up to down [ 498.053734] LNetError: 31503:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 505.036243] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 17:55:52 (1763333752) [ 505.265349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.279747] LNetError: 32209:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 505.293155] LNet: Removed LNI 192.168.204.13@tcp [ 505.312867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.384399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 505.436768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.481918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.584508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 505.672394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.731106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.783608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 505.865751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.917935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.054630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 506.168355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.212463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.267574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 506.352233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.389492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.467186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 506.545779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.590939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.640123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 506.751919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.792588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.887287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 506.994449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.047845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.095028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 507.155808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.177376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.226912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 507.298811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.344320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.382257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 507.418400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.448653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.502095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 512.408433] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 17:56:00 (1763333760) [ 513.770185] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg413-server [ 515.094313] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 17:56:02 (1763333762) [ 516.573304] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg413-server [ 517.919418] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 17:56:05 (1763333765) [ 518.701921] Key type .llcrypt unregistered [ 518.707314] Key type ._llcrypt unregistered [ 522.190616] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 522.292272] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 522.294609] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 522.296701] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 524.705305] Key type ._llcrypt registered [ 524.708613] Key type .llcrypt registered [ 524.858184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 524.874336] alg: No test for adler32 (adler32-zlib) [ 525.840341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.918589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 525.972519] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 526.541949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 526.571409] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 528.198181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 528.219224] LNet: Removed LNI 10.1.2.3@tcp [ 528.344609] LNet: 34326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.353606] LNetError: 34326:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 528.367726] LNet: Removed LNI 192.168.204.13@tcp [ 528.853497] Key type .llcrypt unregistered [ 528.854899] Key type ._llcrypt unregistered [ 536.802126] Key type ._llcrypt registered [ 536.803642] Key type .llcrypt registered [ 537.759824] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 17:56:25 (1763333785) [ 538.133566] Key type .llcrypt unregistered [ 538.135436] Key type ._llcrypt unregistered [ 541.952262] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 542.093664] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 542.095693] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 542.100385] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 544.419266] Key type ._llcrypt registered [ 544.421403] Key type .llcrypt registered [ 544.576064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 544.584485] alg: No test for adler32 (adler32-zlib) [ 545.465985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.531899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 545.568638] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 546.106593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 546.126289] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 548.149331] LNet: 35810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 548.154700] LNetError: 35810:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 548.159693] LNet: Removed LNI 192.168.204.13@tcp [ 548.166659] LNet: Removed LNI 10.1.2.3@tcp [ 548.561429] Key type .llcrypt unregistered [ 548.564466] Key type ._llcrypt unregistered [ 555.899437] Key type ._llcrypt registered [ 555.901518] Key type .llcrypt registered [ 557.063400] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 17:56:44 (1763333804) [ 557.636380] Key type .llcrypt unregistered [ 557.640688] Key type ._llcrypt unregistered [ 561.506439] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 561.739760] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 561.750783] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 561.759155] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 564.163117] Key type ._llcrypt registered [ 564.164639] Key type .llcrypt registered [ 564.312804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 564.324474] alg: No test for adler32 (adler32-zlib) [ 565.221130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.254724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 565.269630] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 566.666024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 566.684855] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 568.498023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 568.510714] LNet: Removed LNI 10.1.2.3@tcp [ 568.521576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 568.544194] LNet: Removed LNI 10.1.2.31@tcp [ 568.562923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 568.589098] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 569.222488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 569.280634] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 570.201983] LNet: 37722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.206587] LNetError: 37722:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 570.210048] LNet: Removed LNI 10.1.2.3@tcp [ 570.221492] LNet: Removed LNI 10.1.2.31@tcp [ 570.692206] Key type .llcrypt unregistered [ 570.694107] Key type ._llcrypt unregistered [ 578.995731] Key type ._llcrypt registered [ 578.997278] Key type .llcrypt registered [ 580.107344] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 17:57:07 (1763333827) [ 580.484716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 580.494596] alg: No test for adler32 (adler32-zlib) [ 581.376409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.433578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 581.468534] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 582.253718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.267125] LNet: 38669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.273874] LNetError: 38669:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 582.286864] LNet: Removed LNI 192.168.204.13@tcp [ 582.313367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.367252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.13@tcp [ 582.391327] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 583.202991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.13@tcp [ 583.249586] LNet: Removed LNI 192.168.204.13@tcp [ 583.302881] LNet: 38805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 583.900510] Key type .llcrypt unregistered [ 583.902579] Key type ._llcrypt unregistered [ 593.305114] Key type ._llcrypt registered [ 593.306769] Key type .llcrypt registered [ 594.561287] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 17:57:22 (1763333842) [ 596.193711] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg413-server [ 597.600757] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 17:57:25 (1763333845) [ 598.107989] Key type .llcrypt unregistered [ 598.109986] Key type ._llcrypt unregistered [ 602.437436] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 602.568984] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 602.570521] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 602.572050] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 605.111762] Key type ._llcrypt registered [ 605.115304] Key type .llcrypt registered [ 605.253187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 605.267861] alg: No test for adler32 (adler32-zlib) [ 606.158984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.220851] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 606.847722] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 608.904699] LNet: 40627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.913722] LNetError: 40627:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 608.916474] LNet: Removed LNI 192.168.204.13@tcp1 [ 608.928886] LNet: Removed LNI 10.1.2.3@tcp2 [ 609.580792] Key type .llcrypt unregistered [ 609.583844] Key type ._llcrypt unregistered [ 621.195172] Key type ._llcrypt registered [ 621.199678] Key type .llcrypt registered [ 622.923583] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 17:57:49 (1763333869) [ 623.468185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 623.480753] alg: No test for adler32 (adler32-zlib) [ 624.417508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.464546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 624.491918] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 625.556351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.569726] LNet: 41592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.577217] LNetError: 41592:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 625.598739] LNet: Removed LNI 192.168.204.13@tcp [ 625.618781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.711683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 625.767886] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 632.691086] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 17:57:59 (1763333879) [ 632.882246] LNet: 42175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.886769] LNetError: 42175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 632.900899] LNet: Removed LNI 192.168.204.13@tcp [ 633.865721] Key type .llcrypt unregistered [ 633.869665] Key type ._llcrypt unregistered [ 639.988910] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 640.114784] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 640.117139] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 640.126441] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 642.808628] Key type ._llcrypt registered [ 642.813471] Key type .llcrypt registered [ 643.050230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 643.065568] alg: No test for adler32 (adler32-zlib) [ 643.973111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.012760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 644.047833] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 644.774438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 644.795447] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 646.584141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.602165] LNet: 42923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 646.607452] LNetError: 42923:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 646.611610] LNet: Removed LNI 192.168.204.13@tcp [ 646.623955] LNet: Removed LNI 10.1.2.3@tcp [ 646.645094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.726323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 646.743936] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 646.749905] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 647.057443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.081669] LNet: Removed LNI 192.168.204.13@tcp [ 647.095599] LNet: Removed LNI 10.1.2.3@tcp [ 647.112899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.165982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 647.183640] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 647.189704] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 647.464274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 647.482216] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 648.350874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.368655] LNet: 43112:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 648.371929] LNet: 43112:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 648.376630] LNetError: 43112:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 648.382691] LNetError: 43112:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 648.387190] LNet: Removed LNI 192.168.204.13@tcp [ 648.392810] LNet: Removed LNI 10.1.2.3@tcp [ 648.412026] LNet: Removed LNI 192.168.204.13@tcp2 [ 648.434201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.491286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 648.517081] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 648.522785] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 648.529314] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 654.160730] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 17:58:21 (1763333901) [ 654.362264] LNet: 43711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.371577] LNetError: 43711:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.376576] LNetError: 43711:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 654.385312] LNet: Removed LNI 192.168.204.13@tcp [ 654.392405] LNet: Removed LNI 10.1.2.3@tcp [ 654.403814] LNet: Removed LNI 192.168.204.13@tcp2 [ 654.888943] Key type .llcrypt unregistered [ 654.890348] Key type ._llcrypt unregistered [ 659.415598] Key type ._llcrypt registered [ 659.417263] Key type .llcrypt registered [ 659.542574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 659.559903] alg: No test for adler32 (adler32-zlib) [ 660.422738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.474265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 660.499225] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 661.156754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.175164] LNet: 44101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.178526] LNetError: 44101:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 661.196560] LNet: Removed LNI 192.168.204.13@tcp [ 661.218265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.292461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 661.317268] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 661.763209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.781400] LNet: 44131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.789411] LNetError: 44131:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 661.808350] LNet: Removed LNI 192.168.204.13@tcp [ 661.823300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.889307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 661.916904] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 662.284049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.326667] LNet: Removed LNI 192.168.204.13@tcp [ 662.355601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.402609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 662.425362] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 662.667792] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 662.833529] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 662.837518] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 662.844200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 665.139568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.157847] LNet: 44239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.167694] LNet: 44239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 665.180847] LNetError: 44239:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.184424] LNetError: 44239:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 665.202787] LNet: Removed LNI 192.168.204.13@tcp [ 665.223344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.280704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 665.321440] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 671.016704] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 17:58:38 (1763333918) [ 671.118826] LNet: 44829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.134606] LNetError: 44829:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.168167] LNet: Removed LNI 192.168.204.13@tcp [ 671.746571] Key type .llcrypt unregistered [ 671.752918] Key type ._llcrypt unregistered [ 676.656033] Key type ._llcrypt registered [ 676.657576] Key type .llcrypt registered [ 676.908933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 676.923961] alg: No test for adler32 (adler32-zlib) [ 677.847895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.909757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 677.939300] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 678.683230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.706484] LNet: 45229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.711532] LNetError: 45229:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.723304] LNet: Removed LNI 192.168.204.13@tcp [ 678.741235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.803390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 678.826499] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 684.422750] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 17:58:52 (1763333932) [ 684.625105] LNet: 45822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.632710] LNetError: 45822:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 684.657792] LNet: Removed LNI 192.168.204.13@tcp [ 685.136767] Key type .llcrypt unregistered [ 685.138675] Key type ._llcrypt unregistered [ 689.284673] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 689.449170] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 689.457105] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 689.463167] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 691.905400] Key type ._llcrypt registered [ 691.907434] Key type .llcrypt registered [ 692.083974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 692.120680] alg: No test for adler32 (adler32-zlib) [ 693.002946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.074164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 693.099510] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 693.777700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.792590] LNet: 46266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.799587] LNetError: 46266:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.811803] LNet: Removed LNI 192.168.204.13@tcp [ 693.829903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.903756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 693.925265] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 699.636096] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 17:59:07 (1763333947) [ 699.744926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.767443] LNet: 46858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.770374] LNetError: 46858:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 699.787434] LNet: Removed LNI 192.168.204.13@tcp [ 699.812932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.961591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 704.920486] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 17:59:12 (1763333952) [ 705.002832] LNet: 47436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.514537] Key type .llcrypt unregistered [ 705.517223] Key type ._llcrypt unregistered [ 709.490020] Key type ._llcrypt registered [ 709.493597] Key type .llcrypt registered [ 709.626150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 709.635774] alg: No test for adler32 (adler32-zlib) [ 710.513916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.564527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 710.580971] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 711.304327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.315719] LNet: 47840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.320287] LNetError: 47840:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.336094] LNet: Removed LNI 192.168.204.13@tcp [ 711.360364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.422318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 711.442966] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 716.473432] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 17:59:24 (1763333964) [ 716.592165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.628113] LNet: 48433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.631252] LNetError: 48433:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 716.676577] LNet: Removed LNI 192.168.204.13@tcp [ 716.708677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.794866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 721.789980] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 17:59:29 (1763333969) [ 723.241313] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 724.670650] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 17:59:32 (1763333972) [ 726.026986] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 727.351465] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 17:59:35 (1763333975) [ 728.618464] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 730.145327] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 17:59:37 (1763333977) [ 731.255991] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 732.667831] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 17:59:40 (1763333980) [ 733.916618] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 735.401612] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 17:59:43 (1763333983) [ 735.527771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.560927] LNet: 50479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.583627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.671752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 735.701380] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 736.456500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.13 [ 736.474996] LNet: Removed LNI 192.168.204.13@tcp [ 741.487979] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 17:59:49 (1763333989) [ 741.582391] LNet: 51181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.588245] LNetError: 51181:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.611359] LNet: Removed LNI 192.168.204.13@tcp [ 742.263779] Key type .llcrypt unregistered [ 742.267284] Key type ._llcrypt unregistered [ 748.506602] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 748.796728] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 748.800039] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 748.802444] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 750.433333] Key type ._llcrypt registered [ 750.434634] Key type .llcrypt registered [ 750.558981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 750.595945] alg: No test for adler32 (adler32-zlib) [ 751.991192] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 759.724617] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 18:00:07 (1763334007) [ 759.860093] LNet: 52076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.876592] LNetError: 52076:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 759.897599] LNet: Removed LNI 10.1.2.3@tcp [ 760.731439] Key type .llcrypt unregistered [ 760.740329] Key type ._llcrypt unregistered [ 766.273040] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 766.457235] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 766.459625] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 766.464183] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 769.115261] Key type ._llcrypt registered [ 769.116876] Key type .llcrypt registered [ 769.260314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 769.272913] alg: No test for adler32 (adler32-zlib) [ 770.202894] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 776.064374] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 18:00:23 (1763334023) [ 776.299073] LNet: 52971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.306280] LNetError: 52971:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 776.320976] LNet: Removed LNI 10.1.2.3@tcp [ 776.928696] Key type .llcrypt unregistered [ 776.930127] Key type ._llcrypt unregistered [ 782.352029] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 782.609930] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 782.613758] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 782.621539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 785.364749] Key type ._llcrypt registered [ 785.366380] Key type .llcrypt registered [ 785.544554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 785.564416] alg: No test for adler32 (adler32-zlib) [ 786.548800] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 792.673665] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 18:00:40 (1763334040) [ 792.756829] LNet: 53853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.774653] LNetError: 53853:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 792.798703] LNet: Removed LNI 10.1.2.3@tcp [ 793.455852] Key type .llcrypt unregistered [ 793.457886] Key type ._llcrypt unregistered [ 798.880271] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 799.135421] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 799.137759] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 799.143553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 801.767549] Key type ._llcrypt registered [ 801.769109] Key type .llcrypt registered [ 801.906912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 801.928888] alg: No test for adler32 (adler32-zlib) [ 802.953779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.066243] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 813.186057] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 18:01:00 (1763334060) [ 813.606468] LNet: 55101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.611742] LNetError: 55101:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 813.625030] LNet: Removed LNI 10.1.2.3@tcp [ 814.219711] Key type .llcrypt unregistered [ 814.223248] Key type ._llcrypt unregistered [ 822.759745] Key type ._llcrypt registered [ 822.761349] Key type .llcrypt registered [ 823.059089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 823.091409] alg: No test for adler32 (adler32-zlib) [ 824.031046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 824.077198] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 830.469268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 831.087591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_interrupt -r 1 [ 831.119253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_interrupt -r 1 [ 831.165155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 831.186843] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 831.264892] LNet: 1 local NIs in recovery (showing 1): 192.168.204.13@tcp [ 831.362075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 831.392619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 831.456180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 831.509659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 831.543367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 831.576865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 831.606992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 831.630386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 831.643603] LNet: 55535:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 831.661692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 831.700553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 831.723702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 831.754808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 831.783068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 831.809666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 831.840075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 832.258623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_dropped -r 1 [ 832.290677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_dropped -r 1 [ 832.317981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 832.332797] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 832.483947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 832.515500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 832.541765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 832.565959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 832.597771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 832.618895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 832.696764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 832.728437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 832.742281] LNet: 55644:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 832.757186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 832.779777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 832.800503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 832.839729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 832.867954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 832.890864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 832.914751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 833.325057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_aborted -r 1 [ 833.353097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_aborted -r 1 [ 833.391463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 833.417956] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 833.588870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 833.610880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 833.638903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 833.668287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 833.696391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 833.725572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 833.767167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 833.803270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 833.817395] LNet: 55753:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 833.833001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 833.851738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 833.884274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 833.917219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 833.942535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 833.972946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 833.999316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 834.446312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_no_route -r 1 [ 834.477699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_no_route -r 1 [ 834.519513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 834.718691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 834.748788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 834.785827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 834.813630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 834.842130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 834.875924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 834.901560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 834.926574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 834.949502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 834.993727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 835.051822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 835.080741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 835.106601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 835.133895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 835.165441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 835.661494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_timeout -r 1 [ 835.715116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_timeout -r 1 [ 835.775659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 835.806734] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 835.811711] LNet: Skipped 1 previous similar message [ 835.995182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 836.019390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 836.052618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 836.076197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 836.126771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 836.177947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 836.219347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 836.259072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 836.293825] LNet: 55972:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 836.318832] LNet: 55972:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 836.349159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 836.400543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 836.423936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 836.461090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 836.484364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 836.541311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 836.585441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 837.120558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_error -r 1 [ 837.157948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_error -r 1 [ 837.207030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 837.433825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 837.466969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 837.491391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 837.554122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 837.584470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 837.629079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 837.651590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 837.700740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 837.764701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 837.812617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 837.847030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 837.885995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 837.924756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 837.945973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 837.979518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 838.249748] LNet: 56121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.253960] LNetError: 56121:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.265947] LNet: Removed LNI 192.168.204.13@tcp [ 838.594919] Key type .llcrypt unregistered [ 838.596704] Key type ._llcrypt unregistered [ 850.979806] Key type ._llcrypt registered [ 850.982528] Key type .llcrypt registered [ 851.866532] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 18:01:39 (1763334099) [ 852.503727] Key type .llcrypt unregistered [ 852.505359] Key type ._llcrypt unregistered [ 856.137348] Key type ._llcrypt registered [ 856.139017] Key type .llcrypt registered [ 856.365872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 856.395484] alg: No test for adler32 (adler32-zlib) [ 857.280936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 857.310632] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 862.874319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 871.092586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.119339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 871.141366] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 872.498892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_interrupt -r 1 [ 872.535528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_interrupt -r 1 [ 872.569376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_interrupt -r 1 [ 872.608979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_interrupt -r 1 [ 872.667460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_interrupt -r 1 [ 872.702659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_interrupt -r 1 [ 872.747168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_interrupt -r 1 [ 872.803202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_interrupt -r 1 [ 872.862222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 872.875711] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 872.883856] LNet: 1 local NIs in recovery (showing 1): 192.168.204.13@tcp [ 873.074374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 873.102189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 873.128589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 873.167235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 873.210598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 873.244661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 873.289071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 873.320778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 873.336358] LNet: 57581:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 873.354045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 873.380150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 873.406291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 873.433490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 873.485378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 873.526294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 873.556474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 874.097336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_dropped -r 1 [ 874.153368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_dropped -r 1 [ 874.227519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_dropped -r 1 [ 874.254529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_dropped -r 1 [ 874.283537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_dropped -r 1 [ 874.345499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_dropped -r 1 [ 874.388870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_dropped -r 1 [ 874.454125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_dropped -r 1 [ 874.521474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 874.556671] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 874.565408] LNet: Skipped 2 previous similar messages [ 874.785630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 874.829909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 874.856862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 874.878486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 874.913920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 874.930850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 874.969941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 874.998939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 875.013422] LNet: 57703:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 875.032356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 875.050231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 875.092605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 875.117250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 875.163337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 875.205719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 875.228332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 875.688505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_aborted -r 1 [ 875.719684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_aborted -r 1 [ 875.748805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_aborted -r 1 [ 875.777599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_aborted -r 1 [ 875.827761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_aborted -r 1 [ 875.851766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_aborted -r 1 [ 875.907308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_aborted -r 1 [ 875.943330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_aborted -r 1 [ 875.992625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 876.037308] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 876.040797] LNet: Skipped 2 previous similar messages [ 876.286784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 876.324233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 876.375713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 876.411474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 876.451395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 876.487625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 876.517425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 876.545584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 876.570866] LNet: 57824:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 876.594200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 876.624290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 876.687555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 876.726075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 876.789310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 876.839022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 876.879439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 877.390844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_no_route -r 1 [ 877.446287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_no_route -r 1 [ 877.482941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_no_route -r 1 [ 877.513099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_no_route -r 1 [ 877.614536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_no_route -r 1 [ 877.670909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_no_route -r 1 [ 877.702776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_no_route -r 1 [ 877.732054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_no_route -r 1 [ 877.779351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 878.035222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 878.068298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 878.093337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 878.126805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 878.149521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 878.185527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 878.212748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 878.243292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 878.265457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 878.296269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 878.317713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 878.341427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 878.375977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 878.406239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 878.451498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 878.991287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_timeout -r 1 [ 879.021536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_timeout -r 1 [ 879.053299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_timeout -r 1 [ 879.094091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_timeout -r 1 [ 879.128486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_timeout -r 1 [ 879.177407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_timeout -r 1 [ 879.237816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_timeout -r 1 [ 879.266591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_timeout -r 1 [ 879.319362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 879.339658] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 879.342460] LNet: Skipped 5 previous similar messages [ 879.566416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 879.596848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 879.621587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 879.659336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 879.686290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 879.734603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 879.771322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 879.821716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 879.844782] LNet: 58066:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 879.855389] LNet: 58066:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 879.873637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 879.895983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 879.943983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 879.995801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 880.031775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 880.074148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 880.120707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 880.684086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_error -r 1 [ 880.759125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_error -r 1 [ 880.821809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_error -r 1 [ 880.868921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_error -r 1 [ 880.900561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_error -r 1 [ 880.931263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_error -r 1 [ 880.978430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_error -r 1 [ 881.023824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_error -r 1 [ 881.055291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 881.250058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 881.274737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 881.316448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 881.352682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 881.415931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 881.450209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 881.491901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 881.530881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 881.552406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 881.593383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 881.643470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 881.684822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 881.725081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 881.777876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 881.836263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 886.003379] LNet: 58263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.010911] LNetError: 58263:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.020884] LNet: Removed LNI 192.168.204.13@tcp [ 886.042290] LNet: Removed LNI 192.168.204.13@tcp1 [ 886.346861] Key type .llcrypt unregistered [ 886.348717] Key type ._llcrypt unregistered [ 899.050154] Key type ._llcrypt registered [ 899.051809] Key type .llcrypt registered [ 900.116031] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 18:02:27 (1763334147) [ 901.097380] Key type .llcrypt unregistered [ 901.105160] Key type ._llcrypt unregistered [ 906.054134] Key type ._llcrypt registered [ 906.055932] Key type .llcrypt registered [ 906.229298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 906.255249] alg: No test for adler32 (adler32-zlib) [ 907.177784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 907.198814] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 913.147250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 913.800843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e remote_dropped -r 1 [ 913.848807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e remote_dropped -r 1 [ 913.881845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 913.906349] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 914.115994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 914.137566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 914.176628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 914.207204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 914.248768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 914.295981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 914.318668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 914.358123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 914.377479] LNet: 59505:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 914.402240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 914.440958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 914.466784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 914.489541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 914.523617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 914.562295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 914.598814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 915.167143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e remote_error -r 1 [ 915.229941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e remote_error -r 1 [ 915.304130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 915.326120] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 915.522723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 915.553816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 915.579782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 915.613043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 915.645872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 915.671507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 915.691352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 915.721246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 915.737479] LNet: 59615:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 915.748525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 915.771627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 915.800668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 915.848546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 915.892762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 915.910799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 915.941612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 916.346770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e remote_timeout -r 1 [ 916.370554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e remote_timeout -r 1 [ 916.396739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 916.416099] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 916.611566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 916.641383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 916.668712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 916.706340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 916.752308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 916.772980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 916.810814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 916.847179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 916.857427] LNet: 59725:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 916.871497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 916.907060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 916.941983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 916.974919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 917.007506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 917.037498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 917.065429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 917.298118] LNet: 59765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.308232] LNetError: 59765:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.323202] LNet: Removed LNI 192.168.204.13@tcp [ 917.657689] Key type .llcrypt unregistered [ 917.659499] Key type ._llcrypt unregistered [ 931.292202] Key type ._llcrypt registered [ 931.293885] Key type .llcrypt registered [ 932.516274] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 18:03:00 (1763334180) [ 933.263802] Key type .llcrypt unregistered [ 933.266852] Key type ._llcrypt unregistered [ 937.708286] Key type ._llcrypt registered [ 937.710954] Key type .llcrypt registered [ 937.913855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.930501] alg: No test for adler32 (adler32-zlib) [ 938.843583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 938.876603] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 944.304987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 949.803385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.829330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 949.853587] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 950.888243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e remote_dropped -r 1 [ 950.910274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e remote_dropped -r 1 [ 950.935163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e remote_dropped -r 1 [ 950.964605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e remote_dropped -r 1 [ 950.990734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e remote_dropped -r 1 [ 951.019393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e remote_dropped -r 1 [ 951.045705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e remote_dropped -r 1 [ 951.080474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e remote_dropped -r 1 [ 951.121355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 951.137452] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 951.146745] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.113@tcp [ 951.309722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 951.327686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 951.355339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 951.386709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 951.409959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 951.473810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 951.514807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 951.543750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 951.613474] LNet: 61223:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 951.628524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 951.654903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 951.678281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 951.696853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 951.723266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 951.756540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 951.789947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 951.812836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 952.126272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e remote_error -r 1 [ 952.155807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e remote_error -r 1 [ 952.179589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e remote_error -r 1 [ 952.237493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e remote_error -r 1 [ 952.276522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e remote_error -r 1 [ 952.306562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e remote_error -r 1 [ 952.346869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e remote_error -r 1 [ 952.389423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e remote_error -r 1 [ 952.428967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 952.448371] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 952.452085] LNet: Skipped 2 previous similar messages [ 952.574649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 952.591427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 952.610805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 952.627545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 952.655061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 952.697959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 952.720630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 952.759128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 952.784603] LNet: 61346:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 952.797162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 952.829447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 952.859738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 952.904127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 952.947914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 952.994707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 953.026421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 953.072819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 953.368252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e remote_timeout -r 1 [ 953.394663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e remote_timeout -r 1 [ 953.430397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e remote_timeout -r 1 [ 953.464426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e remote_timeout -r 1 [ 953.490618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e remote_timeout -r 1 [ 953.520465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e remote_timeout -r 1 [ 953.559894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e remote_timeout -r 1 [ 953.592905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e remote_timeout -r 1 [ 953.617177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 953.627377] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 953.774241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 953.797881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 953.830069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 953.849085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 953.869391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 953.894667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 953.919383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 953.941417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 953.953920] LNet: 61469:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 953.964827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 953.990368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 954.019595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 954.049741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 954.083538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 954.116211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 954.148575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 954.182314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 956.970786] LNet: 61546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.976156] LNetError: 61546:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.985034] LNet: Removed LNI 192.168.204.13@tcp [ 956.995695] LNet: Removed LNI 192.168.204.13@tcp1 [ 957.222710] Key type .llcrypt unregistered [ 957.227776] Key type ._llcrypt unregistered [ 966.932184] Key type ._llcrypt registered [ 966.933806] Key type .llcrypt registered [ 968.081748] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 18:03:35 (1763334215) [ 968.462819] Key type .llcrypt unregistered [ 968.464232] Key type ._llcrypt unregistered [ 972.265223] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 972.383446] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 972.397539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 972.403171] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 974.825153] Key type ._llcrypt registered [ 974.826870] Key type .llcrypt registered [ 974.941164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 974.951899] alg: No test for adler32 (adler32-zlib) [ 975.832930] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 975.869416] LNet: 62625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.875185] LNetError: 62625:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.897076] LNet: Removed LNI 192.168.204.13@tcp [ 976.475344] Key type .llcrypt unregistered [ 976.477915] Key type ._llcrypt unregistered [ 980.737334] Key type ._llcrypt registered [ 980.739015] Key type .llcrypt registered [ 980.882169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 980.896378] alg: No test for adler32 (adler32-zlib) [ 981.788842] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 981.791643] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 981.824304] LNet: 62893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.830256] LNetError: 62893:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.834465] LNet: Removed LNI 192.168.204.13@tcp [ 981.844441] LNet: Removed LNI 10.1.2.3@tcp1 [ 982.403724] Key type .llcrypt unregistered [ 982.406165] Key type ._llcrypt unregistered [ 987.116359] Key type ._llcrypt registered [ 987.117804] Key type .llcrypt registered [ 987.385617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 987.396109] alg: No test for adler32 (adler32-zlib) [ 988.303753] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 988.308552] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 988.356558] LNet: 63163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.363931] LNetError: 63163:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.374973] LNet: Removed LNI 192.168.204.13@tcp [ 988.386537] LNet: Removed LNI 10.1.2.3@tcp [ 988.880788] Key type .llcrypt unregistered [ 988.886755] Key type ._llcrypt unregistered [ 993.042509] Key type ._llcrypt registered [ 993.044297] Key type .llcrypt registered [ 993.254173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 993.279816] alg: No test for adler32 (adler32-zlib) [ 994.185425] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 994.189616] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 994.224884] LNet: 63433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.231673] LNetError: 63433:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.235349] LNet: Removed LNI 192.168.204.13@tcp [ 994.248308] LNet: Removed LNI 10.1.2.3@tcp [ 994.805930] Key type .llcrypt unregistered [ 994.812796] Key type ._llcrypt unregistered [ 998.679195] Key type ._llcrypt registered [ 998.681346] Key type .llcrypt registered [ 998.840971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 998.854623] alg: No test for adler32 (adler32-zlib) [ 999.797310] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 999.801190] LNetError: 63693:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 999.805368] LNetError: Error -100 starting up LNI tcp [ 999.816104] LNet: Removed LNI 192.168.204.13@tcp [ 1000.298391] Key type .llcrypt unregistered [ 1000.300777] Key type ._llcrypt unregistered [ 1008.107775] Key type ._llcrypt registered [ 1008.109146] Key type .llcrypt registered [ 1009.297809] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 18:04:16 (1763334256) [ 1010.063451] Key type .llcrypt unregistered [ 1010.066132] Key type ._llcrypt unregistered [ 1014.749690] Key type ._llcrypt registered [ 1014.751541] Key type .llcrypt registered [ 1014.974767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1014.990450] alg: No test for adler32 (adler32-zlib) [ 1015.932383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1015.976342] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1021.986311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 1022.714849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e network_timeout -r 1 [ 1022.775374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e network_timeout -r 1 [ 1022.801967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 1022.826270] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 1023.064845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1023.104172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1023.136199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1023.136326] LNet: 1 local NIs in recovery (showing 1): 192.168.204.13@tcp [ 1023.166771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1023.192476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.213520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1023.235608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.266365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.281350] LNet: 64942:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1023.303216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1023.336662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1023.366617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1023.395812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1023.429973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1023.465396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1023.734520] LNet: 64980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.738190] LNetError: 64980:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.754069] LNet: Removed LNI 192.168.204.13@tcp [ 1024.102615] Key type .llcrypt unregistered [ 1024.103946] Key type ._llcrypt unregistered [ 1037.477250] Key type ._llcrypt registered [ 1037.480093] Key type .llcrypt registered [ 1037.647849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1037.658212] alg: No test for adler32 (adler32-zlib) [ 1038.543327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1038.578128] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1043.357205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 1047.885887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.917722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1047.939140] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 1048.748702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e network_timeout -r 1 [ 1048.771109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e network_timeout -r 1 [ 1048.794949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e network_timeout -r 1 [ 1048.820549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e network_timeout -r 1 [ 1048.843610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e network_timeout -r 1 [ 1048.870874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e network_timeout -r 1 [ 1048.896038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e network_timeout -r 1 [ 1048.918698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e network_timeout -r 1 [ 1048.939529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 1048.950263] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 1049.065492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1049.088246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1049.109098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1049.130513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1049.150596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1049.174170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1049.197248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1049.219936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1049.232668] LNet: 65852:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1049.245329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1049.267539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1049.290838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1049.310552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1049.330744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1049.346488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1051.885485] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 1051.935191] LNet: 1 local NIs in recovery (showing 1): 192.168.204.13@tcp [ 1052.077886] LNet: 65924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.083512] LNetError: 65924:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.089184] LNet: Removed LNI 192.168.204.13@tcp [ 1052.097762] LNet: Removed LNI 192.168.204.13@tcp1 [ 1052.287576] Key type .llcrypt unregistered [ 1052.290064] Key type ._llcrypt unregistered [ 1059.667352] Key type ._llcrypt registered [ 1059.674758] Key type .llcrypt registered [ 1060.621198] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 18:05:08 (1763334308) [ 1060.868213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1060.877487] alg: No test for adler32 (adler32-zlib) [ 1061.731080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.778479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1061.800560] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1062.361791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1062.382332] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 1063.004722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1063.073295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1063.101492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1063.128569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1063.153741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1063.174783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1063.193078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.13@tcp [ 1063.205089] LNetError: 67019:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.13@tcp to recovery [ 1063.840517] LNet: 1 local NIs in recovery (showing 1): 192.168.204.13@tcp [ 1063.840574] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 1065.890791] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 1066.507023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1069.983391] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 1071.036421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1071.078255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1071.186657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1071.226561] LNet: 67137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.240376] LNetError: 67137:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.248218] LNet: Removed LNI 192.168.204.13@tcp [ 1071.266550] LNet: Removed LNI 192.168.204.13@tcp1 [ 1071.289842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1071.368338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1071.391389] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1071.973567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1071.991287] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 1072.670916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1072.723944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1072.779687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1072.822674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1072.859192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1072.886800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1072.917189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1072.949158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.13@tcp [ 1072.976708] LNetError: 67409:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.13@tcp to recovery [ 1073.377039] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 1076.400906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1079.521218] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 1079.529323] LNet: Skipped 1 previous similar message [ 1084.210981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1084.234452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1084.272804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1090.745255] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 18:05:38 (1763334338) [ 1090.987463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1091.010680] LNet: 68127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.017269] LNetError: 68127:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.021062] LNetError: 68127:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1091.033882] LNet: Removed LNI 192.168.204.13@tcp [ 1091.055160] LNet: Removed LNI 192.168.204.13@tcp1 [ 1091.078830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1091.147306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1091.183813] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1091.868392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1091.886959] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 1092.628513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1092.688967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1092.713311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1092.745282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1092.793756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1092.827418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1092.869881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.13@tcp --health 0 [ 1092.996145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1093.151286] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.13@tcp [ 1093.151876] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 1093.163587] LNet: Skipped 1 previous similar message [ 1108.362656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1108.539024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1108.579445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.612940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.13@tcp --health 500 [ 1108.710159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1115.222849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1115.273949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1115.313728] LNet: 68619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.324766] LNetError: 68619:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.327974] LNetError: 68619:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1116.383418] LNet: Removed LNI 192.168.204.13@tcp [ 1116.404374] LNet: Removed LNI 192.168.204.13@tcp1 [ 1116.438562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1116.476374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1116.492349] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1117.100938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1117.116577] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 1117.885543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1117.925266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1117.983142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1118.019397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1118.048844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1118.080370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1118.124889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1118.147235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.13@tcp --health 0 [ 1119.524488] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 1119.528259] LNet: Skipped 3 previous similar messages [ 1120.436325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1130.377546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1130.409326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1130.444290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1135.755647] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 18:06:23 (1763334383) [ 1136.151448] LNet: 69676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.157339] LNetError: 69676:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.167483] LNetError: 69676:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1136.176249] LNet: Removed LNI 192.168.204.13@tcp [ 1136.189292] LNet: Removed LNI 192.168.204.13@tcp1 [ 1136.705924] Key type .llcrypt unregistered [ 1136.710831] Key type ._llcrypt unregistered [ 1140.740846] Key type ._llcrypt registered [ 1140.746395] Key type .llcrypt registered [ 1140.952122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1140.964714] alg: No test for adler32 (adler32-zlib) [ 1141.852654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1141.877203] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1149.516627] Lustre: DEBUG MARKER: Initial discovery [ 1149.582713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.113@tcp [ 1151.671761] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1151.736313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1151.787243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.113@tcp [ 1151.841675] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 1151.879227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1151.937754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.113@tcp -r 1 -m GET -l 3 [ 1151.995427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.113@tcp -r 1 -m GET -e local_error [ 1152.045625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.113@tcp -r 1 -m PUT -l 6 [ 1152.161195] LNet: 1 local NIs in recovery (showing 1): 192.168.204.13@tcp [ 1153.739475] Lustre: DEBUG MARKER: Force oleg413-client.virtnet to discover 192.168.204.113@tcp (in background) [ 1153.827129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.113@tcp [ 1155.261434] Lustre: DEBUG MARKER: Force oleg413-server to discover 192.168.204.13@tcp [ 1156.831629] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 1157.658881] Lustre: DEBUG MARKER: Wait for 70424 [ 1158.952078] Lustre: DEBUG MARKER: Finished wait on 70424 [ 1158.981919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1159.020549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1159.031642] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 1159.221161] LNet: 70881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.225625] LNetError: 70881:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1159.248775] LNet: Removed LNI 192.168.204.13@tcp [ 1159.626692] Key type .llcrypt unregistered [ 1159.630754] Key type ._llcrypt unregistered [ 1172.248601] Key type ._llcrypt registered [ 1172.265192] Key type .llcrypt registered [ 1173.718655] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 18:07:01 (1763334421) [ 1174.253547] Key type .llcrypt unregistered [ 1174.256337] Key type ._llcrypt unregistered [ 1178.985865] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1179.148240] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1179.150354] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1179.163652] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1181.605607] Key type ._llcrypt registered [ 1181.607316] Key type .llcrypt registered [ 1181.784421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1181.797796] alg: No test for adler32 (adler32-zlib) [ 1182.697667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1182.741867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1182.771210] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1183.415590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1183.432688] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1190.618561] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 18:07:18 (1763334438) [ 1190.732062] LNet: 72936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1190.737421] LNetError: 72936:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.746602] LNet: Removed LNI 192.168.204.13@tcp [ 1190.775548] LNet: Removed LNI 10.1.2.3@tcp [ 1191.248522] Key type .llcrypt unregistered [ 1191.250396] Key type ._llcrypt unregistered [ 1195.325218] Key type ._llcrypt registered [ 1195.326941] Key type .llcrypt registered [ 1195.460656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1195.472049] alg: No test for adler32 (adler32-zlib) [ 1196.330123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1196.377793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1196.394795] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1196.900778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1196.911528] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1197.658663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.13@tcp [ 1198.693373] LNetError: Unexpected error -22 connecting to 192.168.204.13@tcp at host 192.168.204.13:988 [ 1202.718962] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 18:07:30 (1763334450) [ 1202.810194] LNet: 74065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.820739] LNetError: 74065:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.871267] LNet: Removed LNI 192.168.204.13@tcp [ 1203.873056] LNetError: 74065:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1203.899248] LNet: Removed LNI 10.1.2.3@tcp [ 1204.479774] Key type .llcrypt unregistered [ 1204.481478] Key type ._llcrypt unregistered [ 1208.621299] Key type ._llcrypt registered [ 1208.625563] Key type .llcrypt registered [ 1208.821608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1208.841829] alg: No test for adler32 (adler32-zlib) [ 1209.746437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1209.831980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1209.872100] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 1210.428046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1210.453767] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 1211.323488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.13@tcp1 --nid 192.168.204.13@tcp2 [ 1214.004011] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp1 192.168.204.13@tcp2 [ 1214.016756] LNetError: 74906:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.13@tcp1 to 192.168.204.13@tcp2 - Route table may be misconfigured [ 1214.034740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp2 192.168.204.13@tcp1 [ 1218.308461] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 18:07:46 (1763334466) [ 1218.371647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1218.383111] LNet: 75473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1218.389488] LNetError: 75473:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1218.392985] LNet: Removed LNI 192.168.204.13@tcp1 [ 1219.434931] LNet: Removed LNI 192.168.204.13@tcp2 [ 1219.456844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1219.498961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1219.513931] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1220.195664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1220.213255] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 1220.934388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1220.972376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp 192.168.204.13@tcp [ 1220.999311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp1 192.168.204.13@tcp1 [ 1221.027988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.13@tcp -d 192.168.204.13@tcp -e network_timeout [ 1221.075181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -e network_timeout [ 1221.130789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -e network_timeout [ 1221.161537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -e network_timeout [ 1221.212441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1221.234806] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 1221.247949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1221.365907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1221.476946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1221.536094] LNet: 1 local NIs in recovery (showing 1): 192.168.204.13@tcp [ 1226.605376] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 18:07:54 (1763334474) [ 1226.793830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1226.816861] LNet: 76341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.821812] LNetError: 76341:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.825012] LNetError: 76341:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1226.831392] LNet: Removed LNI 192.168.204.13@tcp [ 1227.889982] LNet: Removed LNI 192.168.204.13@tcp1 [ 1227.927800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1228.024688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1228.261505] LNet: 76377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.602803] Key type .llcrypt unregistered [ 1228.606512] Key type ._llcrypt unregistered [ 1236.995278] Key type ._llcrypt registered [ 1236.997831] Key type .llcrypt registered [ 1238.120293] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 18:08:05 (1763334485) [ 1238.554111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1238.568418] alg: No test for adler32 (adler32-zlib) [ 1239.472801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1240.519922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1241.989506] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 18:08:09 (1763334489) [ 1242.122747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1242.153231] LNet: 77470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1242.178515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1242.214734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1242.230360] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1242.786543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1242.814082] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 1243.575342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1243.604398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp1 [ 1243.636752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp1 [ 1248.260855] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 18:08:16 (1763334496) [ 1249.501566] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg413-server [ 1250.701921] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 18:08:18 (1763334498) [ 1252.221432] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg413-server [ 1253.565811] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 18:08:21 (1763334501) [ 1254.940957] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg413-server [ 1256.250057] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 18:08:24 (1763334504) [ 1257.329439] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg413-server [ 1258.390769] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 18:08:26 (1763334506) [ 1259.459160] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg413-server [ 1260.475613] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 18:08:28 (1763334508) [ 1261.474924] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg413-server [ 1262.492194] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 18:08:30 (1763334510) [ 1263.744328] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg413-server [ 1265.054468] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 18:08:32 (1763334512) [ 1266.286969] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg413-server [ 1267.437469] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 18:08:35 (1763334515) [ 1268.817412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg413-server [ 1270.594910] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 18:08:38 (1763334518) [ 1270.702425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1270.730509] LNet: 81339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1270.734101] LNetError: 81339:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1270.736553] LNet: Removed LNI 192.168.204.13@tcp [ 1271.785353] LNet: Removed LNI 192.168.204.13@tcp1 [ 1271.808832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1271.860837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1271.883923] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1272.579726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1272.640373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1272.773279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1272.786626] LNet: 81493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.793867] LNetError: 81493:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.798147] LNetError: 81493:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1272.815652] LNet: Removed LNI 192.168.204.13@tcp [ 1272.839912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1272.910443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1272.933039] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1273.527628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1274.768072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1274.810381] LNet: 81647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1274.821692] LNetError: 81647:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.830725] LNet: Removed LNI 192.168.204.13@tcp [ 1274.849118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1274.896541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1274.922609] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1275.594069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1275.651622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1275.853472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1276.904541] LNet: Removed LNI 192.168.204.13@tcp [ 1276.925301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1276.976631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1277.005634] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1277.664587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1277.855763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1277.879765] LNetError: 81940:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.884477] LNetError: 81940:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1277.901277] LNet: Removed LNI 192.168.204.13@tcp [ 1277.918086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1277.973669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1278.003292] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1278.761825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1278.854651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1279.084880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1279.100615] LNet: 82087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.108286] LNet: 82087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1280.169278] LNet: Removed LNI 192.168.204.13@tcp [ 1280.200353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1280.254894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1280.278610] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1280.813694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1281.105958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1282.150917] LNet: Removed LNI 192.168.204.13@tcp [ 1282.196806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1282.247577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1282.285816] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1282.845722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1282.916225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1283.154449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1283.173415] LNetError: 82380:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1283.179727] LNetError: 82380:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1283.196824] LNet: Removed LNI 192.168.204.13@tcp [ 1283.227220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1283.307732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1283.341634] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1283.973535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1284.234686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1285.289595] LNet: Removed LNI 192.168.204.13@tcp [ 1285.328308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1285.399422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1285.426889] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1286.144115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1286.219288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1286.486841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1287.527551] LNet: Removed LNI 192.168.204.13@tcp [ 1287.553322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1287.623836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1287.646749] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1288.441205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1289.996424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1290.020803] LNet: 82827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.024217] LNet: 82827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1291.048039] LNet: Removed LNI 192.168.204.13@tcp [ 1291.086106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1291.186946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1291.250566] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1292.131972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1292.267546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1293.782838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1293.799440] LNetError: 82983:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.809102] LNetError: 82983:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1293.825917] LNet: Removed LNI 192.168.204.13@tcp [ 1293.862572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1293.968716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1294.009180] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1295.129115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1295.460080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1295.521513] LNet: Removed LNI 192.168.204.13@tcp [ 1295.550477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1295.620443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1295.654849] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1296.442510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1296.513762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1296.547186] LNet: Removed LNI 192.168.204.13@tcp [ 1296.590647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1296.685368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1296.736777] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1297.659554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1303.435198] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 18:09:11 (1763334551) [ 1303.593341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1303.638141] LNet: Removed LNI 192.168.204.13@tcp [ 1303.655080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1303.715989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1303.735759] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/180] [ 1305.003690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1305.042964] LNet: Removed LNI 192.168.204.13@tcp231 [ 1305.062764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1305.116396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1305.144899] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/180] [ 1305.151736] LNet: 84113:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.517542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1305.538675] LNet: Removed LNI 192.168.204.13@tcp231 [ 1305.553102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1305.585280] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/180] [ 1306.686176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1306.707899] LNet: 84257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.712778] LNet: 84257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1306.729989] LNet: Removed LNI 192.168.204.13@tcp231 [ 1306.742409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1306.829351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1306.864836] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/0] [ 1306.880205] LNet: 84265:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1307.160395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1307.189573] LNet: Removed LNI 192.168.204.13@tcp231 [ 1307.203290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1307.239801] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/0] [ 1308.392473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1308.429075] LNet: Removed LNI 192.168.204.13@tcp231 [ 1308.457344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1308.513292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1308.534948] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/60] [ 1308.541500] LNet: 84416:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1308.937155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1308.967724] LNet: Removed LNI 192.168.204.13@tcp231 [ 1308.985802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1309.033831] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/60] [ 1310.052575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1310.083880] LNetError: 84560:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.087496] LNetError: 84560:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1310.103552] LNet: Removed LNI 192.168.204.13@tcp231 [ 1310.124230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1310.192025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1310.229445] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/180] [ 1310.649389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1310.687663] LNet: Removed LNI 192.168.204.13@tcp231 [ 1310.702123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1310.730580] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/180] [ 1317.025722] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 18:09:24 (1763334564) [ 1317.198981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1317.237224] LNet: Removed LNI 192.168.204.13@tcp231 [ 1317.261734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1317.329156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1317.361607] LNet: Added LNI 192.168.204.13@tcp232 [8/256/0/180] [ 1318.029187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1318.067440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1318.092661] LNet: Removed LNI 192.168.204.13@tcp232 [ 1318.114356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1318.146482] LNet: Added LNI 192.168.204.13@tcp232 [8/256/0/180] [ 1318.807022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1318.836654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1318.878492] LNet: Removed LNI 192.168.204.13@tcp232 [ 1318.902773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1318.941652] LNet: Added LNI 192.168.204.13@tcp232 [8/256/0/180] [ 1319.497586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1319.537721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1319.566881] LNet: Removed LNI 192.168.204.13@tcp232 [ 1319.578948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1319.610524] LNet: Added LNI 192.168.204.13@tcp232 [8/256/0/180] [ 1320.236452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1320.274613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1320.298847] LNet: Removed LNI 192.168.204.13@tcp232 [ 1320.315745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1325.079981] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 18:09:32 (1763334572) [ 1326.108121] Key type .llcrypt unregistered [ 1326.117942] Key type ._llcrypt unregistered [ 1330.903412] Key type ._llcrypt registered [ 1330.913684] Key type .llcrypt registered [ 1331.148813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1331.166252] alg: No test for adler32 (adler32-zlib) [ 1332.081759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1332.111949] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1338.480996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 1345.261591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1345.321797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1345.352772] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 1346.678666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_interrupt -r 2 [ 1346.704843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_interrupt -r 2 [ 1346.729374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_interrupt -r 2 [ 1346.767138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_interrupt -r 2 [ 1346.796323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_interrupt -r 2 [ 1346.821666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_interrupt -r 2 [ 1346.850328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_interrupt -r 2 [ 1346.886299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_interrupt -r 2 [ 1346.908568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp 192.168.204.113@tcp [ 1346.933966] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 1346.937349] LNet: 1 local NIs in recovery (showing 1): 192.168.204.13@tcp [ 1347.167680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1347.217336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1347.250444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1347.310845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1347.351757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1347.378310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1347.421322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1347.460553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1347.482550] LNet: 87064:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1347.505844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1347.548143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1347.576662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1347.599963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1347.642252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1347.666435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1348.173643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_dropped -r 2 [ 1348.218834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_dropped -r 2 [ 1348.244267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_dropped -r 2 [ 1348.282634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_dropped -r 2 [ 1348.321571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_dropped -r 2 [ 1348.349931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_dropped -r 2 [ 1348.378238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_dropped -r 2 [ 1348.403975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_dropped -r 2 [ 1348.442995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp 192.168.204.113@tcp [ 1348.464903] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 1348.670031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1348.701071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1348.729860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1348.769808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1348.790936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1348.830847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1348.855612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1348.884324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1348.897691] LNet: 87184:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1348.921776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1348.952746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1348.987961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1349.022617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1349.060557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.085317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1349.640624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_aborted -r 2 [ 1349.663755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_aborted -r 2 [ 1349.689217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_aborted -r 2 [ 1349.724147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_aborted -r 2 [ 1349.750967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_aborted -r 2 [ 1349.789467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_aborted -r 2 [ 1349.831477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_aborted -r 2 [ 1349.882340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_aborted -r 2 [ 1349.909562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp 192.168.204.113@tcp [ 1349.920481] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 1350.110129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1350.130219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1350.152515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1350.173442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1350.204906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1350.225609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1350.246613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1350.267152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1350.291570] LNet: 87304:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1350.311931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1350.345120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1350.369411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1350.394308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1350.418865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1350.443661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1350.899940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_no_route -r 2 [ 1350.931886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_no_route -r 2 [ 1350.975040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_no_route -r 2 [ 1351.013622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_no_route -r 2 [ 1351.062853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_no_route -r 2 [ 1351.117296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_no_route -r 2 [ 1351.148993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_no_route -r 2 [ 1351.184036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_no_route -r 2 [ 1351.221583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp 192.168.204.113@tcp [ 1351.442038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1351.464924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1351.484799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1351.510506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1351.545177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1351.562703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1351.585498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1351.615886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1351.645931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1351.673525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1351.695941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1351.726811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1351.754560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1351.788350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1352.272704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_timeout -r 2 [ 1352.310786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_timeout -r 2 [ 1352.338851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_timeout -r 2 [ 1352.356518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_timeout -r 2 [ 1352.404904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_timeout -r 2 [ 1352.439259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_timeout -r 2 [ 1352.492124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_timeout -r 2 [ 1352.529769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_timeout -r 2 [ 1352.581593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp 192.168.204.113@tcp [ 1352.599722] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 1352.611614] LNet: Skipped 1 previous similar message [ 1352.815660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1352.853656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1352.895439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1352.924371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1352.954135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1352.984651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1353.022512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1353.041635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1353.052296] LNet: 87545:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1353.058087] LNet: 87545:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 1353.073446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1353.104591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1353.140409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1353.174683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1353.226614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1353.259720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1353.678913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e remote_dropped -r 2 [ 1353.727491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e remote_dropped -r 2 [ 1353.753163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e remote_dropped -r 2 [ 1353.791668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e remote_dropped -r 2 [ 1353.831770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e remote_dropped -r 2 [ 1353.871766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e remote_dropped -r 2 [ 1353.904759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e remote_dropped -r 2 [ 1353.941390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e remote_dropped -r 2 [ 1353.974457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp 192.168.204.113@tcp [ 1353.992406] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.113@tcp [ 1354.181181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1354.216238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1354.251500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1354.274943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1354.302482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1354.341101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1354.393109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1354.424225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1354.447203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1354.477772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1354.513953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1354.574792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1354.598589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1354.632380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1359.457470] LNet: 87738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.468555] LNetError: 87738:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.479791] LNet: Removed LNI 192.168.204.13@tcp [ 1360.481838] LNetError: 87738:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.509574] LNet: Removed LNI 192.168.204.13@tcp1 [ 1361.002616] Key type .llcrypt unregistered [ 1361.004481] Key type ._llcrypt unregistered [ 1373.409719] Key type ._llcrypt registered [ 1373.411397] Key type .llcrypt registered [ 1374.439806] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 18:10:22 (1763334622) [ 1374.698922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1374.709888] alg: No test for adler32 (adler32-zlib) [ 1375.659318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1375.718991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1375.747759] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1376.509255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1376.541532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.13@tcp [ 1376.560565] LNetError: 88724:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.13@tcp to recovery [ 1376.636786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.13@tcp [ 1376.730510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1382.160713] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 18:10:29 (1763334629) [ 1382.285663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1382.309805] LNet: 89310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.323434] LNetError: 89310:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1382.381761] LNet: Removed LNI 192.168.204.13@tcp [ 1382.393250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.430709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1382.461684] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1383.023733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1383.040742] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 1383.825942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.13@tcp [ 1383.853357] LNetError: 89579:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.13@tcp to recovery [ 1384.481342] LNet: 1 local NIs in recovery (showing 1): 192.168.204.13@tcp [ 1384.484340] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 1385.072439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.13@tcp [ 1392.561977] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 18:10:40 (1763334640) [ 1394.326436] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg413-server [ 1395.849593] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 18:10:43 (1763334643) [ 1395.958374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1395.977909] LNet: 90531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1395.984110] LNetError: 90531:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1395.990106] LNet: Removed LNI 192.168.204.13@tcp [ 1396.008545] LNet: Removed LNI 192.168.204.13@tcp1 [ 1396.025720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.069865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1396.087284] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1396.631236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1396.675792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1396.716622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1396.795149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1396.839371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1396.866243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1401.569469] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 18:10:49 (1763334649) [ 1401.664022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1401.693384] LNet: 91256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.699249] LNetError: 91256:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.702462] LNetError: 91256:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1401.724963] LNet: Removed LNI 192.168.204.13@tcp [ 1401.741737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1401.783766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1401.804089] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1407.544562] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 18:10:55 (1763334655) [ 1407.658448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1407.686512] LNet: 91956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1407.700343] LNetError: 91956:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1407.715963] LNet: Removed LNI 192.168.204.13@tcp [ 1407.747360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1407.823595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1407.853751] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1414.057835] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 18:11:01 (1763334661) [ 1415.566036] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1417.194817] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 18:11:04 (1763334664) [ 1417.714350] LNet: 92997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1417.718973] LNetError: 92997:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1417.729439] LNet: Removed LNI 192.168.204.13@tcp [ 1418.367416] Key type .llcrypt unregistered [ 1418.371895] Key type ._llcrypt unregistered [ 1423.557710] Key type ._llcrypt registered [ 1423.559392] Key type .llcrypt registered [ 1423.738459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1423.755026] alg: No test for adler32 (adler32-zlib) [ 1424.671698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1424.702966] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1430.631315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 1433.682472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.113@tcp [ 1434.004083] LNet: 93408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.008913] LNetError: 93408:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.020528] LNet: Removed LNI 192.168.204.13@tcp [ 1434.296827] Key type .llcrypt unregistered [ 1434.299575] Key type ._llcrypt unregistered [ 1443.137573] Key type ._llcrypt registered [ 1443.141638] Key type .llcrypt registered [ 1444.231927] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1445.866577] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1447.445812] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 18:11:35 (1763334695) [ 1447.840390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1447.867452] alg: No test for adler32 (adler32-zlib) [ 1448.833990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1448.924221] LNet: 94491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.607988] Key type .llcrypt unregistered [ 1449.609704] Key type ._llcrypt unregistered [ 1454.974877] Key type ._llcrypt registered [ 1454.976565] Key type .llcrypt registered [ 1455.217090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1455.240334] alg: No test for adler32 (adler32-zlib) [ 1456.240661] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1456.242535] LNet: 94743:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1456.322502] LNetError: 94755:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.15@tcp is being used as a gateway but routing feature is not turned on [ 1456.334122] LNetError: 94755:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.15@tcp has gone from up to down [ 1456.937256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1456.989245] LNet: 94801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.016932] LNetError: 94801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.055671] LNet: Removed LNI 192.168.204.13@tcp [ 1457.079461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.146996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1457.180616] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1457.188815] LNet: 94809:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1457.204424] LNet: 94809:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1457.222494] LNetError: 94809:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.204.14@tcp is a gateway. Can not delete it [ 1457.229327] LNetError: 94809:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.204.14@tcp is a router. Can not be deleted [ 1457.252621] LNet: 94809:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1457.261967] LNetError: 94805:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.17@tcp is being used as a gateway but routing feature is not turned on [ 1457.287805] LNetError: 94805:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 1457.322203] LNetError: 94805:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.17@tcp has gone from up to down [ 1457.351399] LNetError: 94805:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1463.535597] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 18:11:51 (1763334711) [ 1465.342479] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg413-server [ 1467.243834] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 18:11:54 (1763334714) [ 1468.879594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg413-server [ 1470.833742] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 18:11:58 (1763334718) [ 1472.957530] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 1474.839195] Key type lgssc registered [ 1476.030321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1492.588220] Key type lgssc unregistered [ 1492.920835] LNet: 97226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.935462] LNetError: 97226:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.954139] LNet: Removed LNI 192.168.204.13@tcp [ 1493.470068] Key type .llcrypt unregistered [ 1493.471852] Key type ._llcrypt unregistered [ 1505.606254] Key type ._llcrypt registered [ 1505.607949] Key type .llcrypt registered [ 1506.881703] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 18:12:34 (1763334754) [ 1507.290489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1507.318240] alg: No test for adler32 (adler32-zlib) [ 1508.263034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1508.343722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1508.408560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1508.450541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1508.496526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1514.201665] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 18:12:41 (1763334761) [ 1514.275964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1514.308980] LNet: 98623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1514.356189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1514.440203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1514.462166] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1515.297846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1515.318739] LNet: 98774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1515.324197] LNetError: 98774:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1515.333759] LNet: Removed LNI 192.168.204.13@tcp [ 1515.361672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1515.432640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1515.455742] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1515.758535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1515.784106] LNet: Removed LNI 192.168.204.13@tcp [ 1515.803686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1515.891424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1515.919356] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1522.449879] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 18:12:50 (1763334770) [ 1522.614490] LNet: 99388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1522.617758] LNet: 99388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1522.629238] LNetError: 99388:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1522.640232] LNetError: 99388:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1522.712053] LNet: Removed LNI 192.168.204.13@tcp [ 1523.128153] Key type .llcrypt unregistered [ 1523.130062] Key type ._llcrypt unregistered [ 1527.771813] Key type ._llcrypt registered [ 1527.774607] Key type .llcrypt registered [ 1527.936790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1527.962969] alg: No test for adler32 (adler32-zlib) [ 1528.831479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.889699] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1528.893145] LNet: *** cfs_fail_loc=e003, val=100*** [ 1529.502067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1529.516783] LNet: *** cfs_fail_loc=e003, val=100*** [ 1529.518928] LNet: Skipped 2 previous similar messages [ 1529.525147] Ping reply received with mapping: [ 1529.526832] found metadata reply NI[192.168.204.13@tcp] [ 1529.530481] 0d cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1529.532407] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1529.534374] 5a 5a 5a 5a 5a 5a 5a 5a [ 1529.701544] LNet: 99786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.709566] LNetError: 99786:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1529.724110] LNet: Removed LNI 192.168.204.13@tcp [ 1530.239111] Key type .llcrypt unregistered [ 1530.242206] Key type ._llcrypt unregistered [ 1538.347380] Key type ._llcrypt registered [ 1538.349020] Key type .llcrypt registered [ 1539.384285] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 18:13:07 (1763334787) [ 1539.845430] Key type .llcrypt unregistered [ 1539.847186] Key type ._llcrypt unregistered [ 1544.036380] Key type ._llcrypt registered [ 1544.037975] Key type .llcrypt registered [ 1544.208214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1544.233957] alg: No test for adler32 (adler32-zlib) [ 1545.257609] LNetError: 100821:0:(api-ni.c:2725:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1545.557450] Key type .llcrypt unregistered [ 1545.559058] Key type ._llcrypt unregistered [ 1549.552263] Key type ._llcrypt registered [ 1549.553795] Key type .llcrypt registered [ 1550.575863] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 18:13:18 (1763334798) [ 1551.008331] Key type .llcrypt unregistered [ 1551.010963] Key type ._llcrypt unregistered [ 1555.427877] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1555.430442] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1557.871587] Key type ._llcrypt registered [ 1557.872880] Key type .llcrypt registered [ 1558.047082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1558.058702] alg: No test for adler32 (adler32-zlib) [ 1558.972995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.035887] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1566.647259] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 18:13:34 (1763334814) [ 1566.782780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1566.812148] LNet: 102590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1566.825108] LNetError: 102590:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1566.860225] LNet: Removed LNI 10.1.2.3@tcp [ 1566.887363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.689823] Key type .llcrypt unregistered [ 1567.691771] Key type ._llcrypt unregistered [ 1572.872858] Key type ._llcrypt registered [ 1572.874888] Key type .llcrypt registered [ 1573.063412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1573.073985] alg: No test for adler32 (adler32-zlib) [ 1574.025845] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1574.080665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1575.394729] LNet: 102885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.401973] LNetError: 102885:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.422638] LNet: Removed LNI 192.168.204.13@tcp [ 1576.199365] Key type .llcrypt unregistered [ 1576.201215] Key type ._llcrypt unregistered [ 1585.740417] Key type ._llcrypt registered [ 1585.741864] Key type .llcrypt registered [ 1587.112714] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 18:13:54 (1763334834) [ 1587.753361] Key type .llcrypt unregistered [ 1587.758516] Key type ._llcrypt unregistered [ 1593.106347] Key type ._llcrypt registered [ 1593.108113] Key type .llcrypt registered [ 1593.389556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1593.411495] alg: No test for adler32 (adler32-zlib) [ 1599.906218] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 18:14:07 (1763334847) [ 1600.073940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.123506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.184598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1600.208832] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1600.831414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1600.854495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1600.872519] LNet: Removed LNI 192.168.204.13@tcp [ 1601.052994] LNet: 104620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.467907] Key type .llcrypt unregistered [ 1601.472163] Key type ._llcrypt unregistered [ 1610.482577] Key type ._llcrypt registered [ 1610.484324] Key type .llcrypt registered [ 1611.611966] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 18:14:19 (1763334859) [ 1612.121728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1612.137436] alg: No test for adler32 (adler32-zlib) [ 1613.071693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.122595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1613.144074] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1613.821920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1613.854475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.13@tcp [ 1613.870836] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1613.874694] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1613.877177] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1613, 0, 0 [ 1613.882985] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1613.887054] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1613.892306] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1613.895322] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1613, 0, 0 [ 1613.902035] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1613.905509] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1613.908220] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1613.911197] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1613, 0, 1663 [ 1613.914740] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1613.918299] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1613.923289] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1613.925475] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1613, 0, 1663 [ 1613.932875] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1618.957977] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 18:14:26 (1763334866) [ 1619.635745] LNet: 106156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1619.646853] LNetError: 106156:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1619.675034] LNet: Removed LNI 192.168.204.13@tcp [ 1620.329665] Key type .llcrypt unregistered [ 1620.331154] Key type ._llcrypt unregistered [ 1625.575921] Key type ._llcrypt registered [ 1625.577898] Key type .llcrypt registered [ 1625.785675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1625.793602] alg: No test for adler32 (adler32-zlib) [ 1626.738433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1626.783071] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1633.436557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 1640.575916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.608787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1640.628740] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 1641.423239] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1641.483142] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1641.496824] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1641.500220] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1643.855344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1643.872154] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1653.716821] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 18:15:01 (1763334901) [ 1653.825929] LNet: 107652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.854762] LNetError: 107652:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1653.871102] LNet: Removed LNI 192.168.204.13@tcp [ 1654.879172] LNet: Removed LNI 192.168.204.13@tcp1 [ 1654.881819] LNetError: 107652:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1654.906268] LNetError: 107652:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1654.936742] LNet: Removed LNI 10.1.2.3@tcp99 [ 1655.671712] Key type .llcrypt unregistered [ 1655.673481] Key type ._llcrypt unregistered [ 1661.567102] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1661.569538] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1664.139442] Key type ._llcrypt registered [ 1664.141185] Key type .llcrypt registered [ 1664.386428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1664.415769] alg: No test for adler32 (adler32-zlib) [ 1665.343177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.416995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1665.442269] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1666.042501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1666.066319] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1668.159837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1668.195893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1668.231440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1668.250662] LNetError: 108401:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1668.274486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1668.312071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1668.581230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1668.670194] LNet: 108421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.677288] LNetError: 108421:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.689044] LNet: Removed LNI 192.168.204.13@tcp [ 1668.711619] LNet: Removed LNI 10.1.2.3@tcp [ 1669.362725] Key type .llcrypt unregistered [ 1669.365889] Key type ._llcrypt unregistered [ 1680.498797] Key type ._llcrypt registered [ 1680.516586] Key type .llcrypt registered [ 1682.132917] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 18:15:29 (1763334929) [ 1682.649850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1682.661706] alg: No test for adler32 (adler32-zlib) [ 1683.627891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.669808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1683.695333] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1684.342617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 1684.408154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg413-client@tcp [ 1689.931889] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 18:15:37 (1763334937) [ 1690.053267] LNet: 109944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.058126] LNetError: 109944:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.088899] LNet: Removed LNI 192.168.204.13@tcp [ 1690.709748] Key type .llcrypt unregistered [ 1690.714370] Key type ._llcrypt unregistered [ 1695.487304] Key type ._llcrypt registered [ 1695.488532] Key type .llcrypt registered [ 1695.674569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1695.686876] alg: No test for adler32 (adler32-zlib) [ 1696.583510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1696.604942] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1696.641500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1702.470982] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 18:15:50 (1763334950) [ 1703.753773] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1705.423769] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 18:15:53 (1763334953) [ 1706.710578] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1708.195685] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 18:15:55 (1763334955) [ 1708.319415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.361031] LNet: 111364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.374901] LNetError: 111364:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1708.397099] LNet: Removed LNI 192.168.204.13@tcp [ 1708.417752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.465743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1708.490027] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1709.261822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1709.309920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1715.597661] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 18:16:03 (1763334963) [ 1715.821962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.856216] LNet: 112072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.871644] LNetError: 112072:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.913728] LNet: Removed LNI 192.168.204.13@tcp [ 1715.942981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.024684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1716.079358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1716.781375] Key type .llcrypt unregistered [ 1716.786826] Key type ._llcrypt unregistered [ 1726.635402] Key type ._llcrypt registered [ 1726.637545] Key type .llcrypt registered [ 1728.013843] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 18:16:15 (1763334975) [ 1728.362110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1728.383454] alg: No test for adler32 (adler32-zlib) [ 1729.340642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.410436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1729.464468] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1730.201417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1730.249914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1735.623944] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 18:16:23 (1763334983) [ 1735.860274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.878736] LNet: 113625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.884261] LNetError: 113625:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1735.896924] LNet: Removed LNI 192.168.204.13@tcp [ 1735.926574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.958660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1735.984944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1741.399062] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 18:16:29 (1763334989) [ 1741.616047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.643793] LNet: 114206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.657321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.709497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1742.008828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1742.050535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1742.103051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1742.405278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1742.436207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1742.503505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.535320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.601965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1742.886428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1742.922703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1742.966549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1743.274044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1743.315272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1749.200223] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 18:16:36 (1763334996) [ 1749.313572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.338212] LNet: 114831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.350148] LNet: 114831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1749.385978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.461322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1749.514768] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1750.138802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1750.159564] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 1750.926484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1750.984278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1751.147940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.168683] LNetError: 115118:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1752.226654] LNet: Removed LNI 192.168.204.13@tcp [ 1752.228498] LNetError: 115118:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1752.244463] LNet: Removed LNI 192.168.204.13@tcp2 [ 1752.307787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.438582] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1752.453854] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 1752.456744] LNet: 115124:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1753.178863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1753.479794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.511289] LNet: 115161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.521276] LNet: 115161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1754.591305] LNet: Removed LNI 192.168.204.13@tcp [ 1754.594130] LNetError: 115161:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.602812] LNetError: 115161:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1754.627096] LNet: Removed LNI 192.168.204.13@tcp2 [ 1754.656056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.709864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1754.730262] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1755.397618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1755.414263] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 1756.180135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1756.223983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1756.265814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1756.506956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.599295] LNet: Removed LNI 192.168.204.13@tcp [ 1758.630698] LNet: Removed LNI 192.168.204.13@tcp2 [ 1758.652282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.706765] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1758.713610] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 1758.718616] LNet: 115452:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1759.276108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1759.458344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.478854] LNet: 115489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.483449] LNet: 115489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1759.490204] LNetError: 115489:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.493811] LNetError: 115489:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1759.497954] LNet: Removed LNI 192.168.204.13@tcp [ 1760.554269] LNet: Removed LNI 192.168.204.13@tcp2 [ 1760.583957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.670032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1760.711835] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1761.378558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1761.391700] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 1762.206890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1762.253860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1762.438628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.490177] LNet: Removed LNI 192.168.204.13@tcp [ 1763.503866] LNet: Removed LNI 192.168.204.13@tcp2 [ 1763.539854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.619415] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1763.635172] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 1763.652288] LNet: 115775:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1763.968840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1764.122673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.215418] LNet: Removed LNI 192.168.204.13@tcp [ 1765.238366] LNet: Removed LNI 192.168.204.13@tcp2 [ 1765.269477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.333705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1765.355755] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1766.076502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1766.103335] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 1766.997631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1767.067751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1767.123891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1767.387081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.415402] LNet: Removed LNI 192.168.204.13@tcp [ 1768.424282] LNetError: 116096:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1768.427229] LNetError: 116096:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1768.453436] LNet: Removed LNI 192.168.204.13@tcp2 [ 1768.482662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.558763] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1768.572036] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 1768.579112] LNet: 116103:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1769.015740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1769.193431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.212913] LNet: 116140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.218769] LNet: 116140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1769.226555] LNet: Removed LNI 192.168.204.13@tcp [ 1770.287032] LNet: Removed LNI 192.168.204.13@tcp2 [ 1770.305608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.367951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1770.386818] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1770.951523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1770.979126] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 1771.677566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1771.707872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1771.761706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1772.012655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.090219] LNet: Removed LNI 192.168.204.13@tcp [ 1773.107861] LNet: Removed LNI 192.168.204.13@tcp2 [ 1773.150878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.242634] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1773.251434] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 1773.254454] LNet: 116442:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1773.685795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1773.975161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1775.007577] LNet: Removed LNI 192.168.204.13@tcp [ 1775.016936] LNet: Removed LNI 192.168.204.13@tcp2 [ 1775.032878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.085277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1775.134204] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1775.853721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1775.878204] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 1776.789310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1776.817630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1776.842924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1776.893538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1777.169780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.207154] LNet: Removed LNI 192.168.204.13@tcp [ 1779.240256] LNet: Removed LNI 192.168.204.13@tcp2 [ 1779.258630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.317596] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1779.321912] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 1779.885650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1787.422659] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1788.921590] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 18:17:16 (1763335036) [ 1789.059509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1789.138564] LNet: 117555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.141953] LNet: 117555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1789.165698] LNetError: 117555:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1789.175696] LNetError: 117555:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1789.179506] LNet: Removed LNI 192.168.204.13@tcp [ 1790.186327] LNet: Removed LNI 192.168.204.13@tcp2 [ 1790.223669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.568209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1790.601275] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1791.387472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1791.423632] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1792.401919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1792.425369] LNet: Removed LNI 192.168.204.13@tcp [ 1792.438068] LNet: Removed LNI 10.1.2.3@tcp [ 1792.464159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.532819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1792.561429] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1792.569713] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1793.208379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1793.248734] LNet: Removed LNI 192.168.204.13@tcp [ 1793.259373] LNet: Removed LNI 10.1.2.3@tcp [ 1799.042824] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 18:17:26 (1763335046) [ 1799.176055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.282538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.427315] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1799.516275] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1799.518774] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1799.521386] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1801.831880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1801.862420] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1802.531093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1802.560711] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1804.674484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 1804.728712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1804.808134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.13@tcp -m PUT -e local_timeout -r 1 [ 1804.890433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1804.962542] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1804.971976] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1805.183385] LNetError: 118501:0:(lib-move.c:4947:lnet_parse()) 192.168.204.13@tcp, src 192.168.204.13@tcp: Dropping PUT (error -108 looking up sender) [ 1807.263434] LNet: Removed LNI 192.168.204.13@tcp [ 1807.279342] LNet: Removed LNI 10.1.2.3@tcp [ 1807.727587] Key type .llcrypt unregistered [ 1807.729186] Key type ._llcrypt unregistered [ 1818.840637] Key type ._llcrypt registered [ 1818.841936] Key type .llcrypt registered [ 1820.116330] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 18:17:47 (1763335067) [ 1820.434540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1820.449915] alg: No test for adler32 (adler32-zlib) [ 1821.396355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.685512] LNet: 119806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.212458] Key type .llcrypt unregistered [ 1822.215077] Key type ._llcrypt unregistered [ 1827.757331] Key type ._llcrypt registered [ 1827.759808] Key type .llcrypt registered [ 1827.984682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1828.002985] alg: No test for adler32 (adler32-zlib) [ 1828.936381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1828.974779] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1834.751577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 1834.974895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1835.549145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.113@tcp [ 1835.788343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1835.810094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1835.832703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1835.871908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1835.905671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1835.951357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1835.989615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1836.073578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1836.119852] LNet: 120251:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1836.143044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1836.171238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1836.198196] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.113@tcp [ 1836.207716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1836.245273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1836.279377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1836.326256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1837.030767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1837.312243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1837.367978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1837.421806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1837.446858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1837.479542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1837.509634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1837.570852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1837.625326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1837.644207] LNet: 120369:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1837.668949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1837.712537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1837.766196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1837.800809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1837.839871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1837.858931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1838.561978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.113@tcp [ 1838.887578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1838.909690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1838.952493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1838.972248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1839.002966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1839.036847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1839.077696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1839.112710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1839.122917] LNet: 120486:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1839.155903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1839.190925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1839.232330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1839.267343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1839.314549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1839.372684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1840.212966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1840.468891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1840.522905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1840.570259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1840.628915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1840.691992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1840.732174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1840.790402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1840.841375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1840.872453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1840.895989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1840.918095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1840.951497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1840.977672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1841.026327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1841.381189] LNet: 120648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.392707] LNetError: 120648:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.422669] LNet: Removed LNI 192.168.204.13@tcp [ 1841.916685] Key type .llcrypt unregistered [ 1841.920340] Key type ._llcrypt unregistered [ 1855.534803] Key type ._llcrypt registered [ 1855.536489] Key type .llcrypt registered [ 1856.880607] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 18:18:24 (1763335104) [ 1857.322347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1857.332513] alg: No test for adler32 (adler32-zlib) [ 1858.257273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.696760] LNet: 121548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.415682] Key type .llcrypt unregistered [ 1859.422791] Key type ._llcrypt unregistered [ 1864.910242] Key type ._llcrypt registered [ 1864.914292] Key type .llcrypt registered [ 1865.201804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1865.245896] alg: No test for adler32 (adler32-zlib) [ 1866.233378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1866.273675] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1873.782713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 1882.658854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1882.699335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1882.727942] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 1883.666881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1884.341565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.113@tcp [ 1884.583573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1884.652223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1884.676135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1884.703328] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.113@tcp [ 1884.713904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1884.745656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1884.782872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1884.813528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1884.836708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1884.857440] LNet: 122196:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1884.897831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1884.924948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1884.961123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1884.983214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1885.017615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1885.078161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1885.806792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1886.074781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1886.130130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1886.184954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1886.215358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1886.241459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1886.282540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1886.326425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1886.348879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1886.377343] LNet: 122313:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1886.410996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1886.474470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1886.508497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1886.540703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1886.601166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1886.673583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1887.457302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.113@tcp [ 1887.654126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1887.697155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1887.735693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1887.784680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1887.826259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1887.854171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1887.902173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1887.939092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1887.961635] LNet: 122430:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1887.978693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1888.013788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1888.055260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1888.098620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1888.145314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1888.178608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1888.757465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1888.998684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1889.052378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1889.108895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1889.162733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1889.212873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1889.257437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1889.283398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1889.321295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1889.369191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1889.411665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1889.468279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1889.490799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1889.513133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1889.545816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1893.920274] LNet: 122627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.927468] LNetError: 122627:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.936139] LNet: Removed LNI 192.168.204.13@tcp [ 1893.951464] LNet: Removed LNI 192.168.204.13@tcp1 [ 1894.344313] Key type .llcrypt unregistered [ 1894.348940] Key type ._llcrypt unregistered [ 1909.592633] Key type ._llcrypt registered [ 1909.594374] Key type .llcrypt registered [ 1910.709488] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1713 sec ========== 18:19:18 (1763335158) [ 1911.368512] Key type .llcrypt unregistered [ 1911.370480] Key type ._llcrypt unregistered [ 1920.968611] Key type ._llcrypt registered [ 1920.970932] Key type .llcrypt registered [ 1921.425921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1921.437065] alg: No test for adler32 (adler32-zlib) [ 1922.617872] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 1922.969537] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 1924.672147] Key type lgssc registered [ 1925.690475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1979.305411] Lustre: Mounted lustre-client [ 1983.793710] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1995.536828] LustreError: 125852:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff913d02aa8800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1995.550824] LustreError: 125852:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1995.585483] Lustre: Unmounted lustre-client [ 2013.187860] Key type lgssc unregistered [ 2013.447853] LNet: 126332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.466570] LNetError: 126332:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2013.500247] LNet: Removed LNI 192.168.204.13@tcp [ 2014.372208] Key type .llcrypt unregistered [ 2014.377849] Key type ._llcrypt unregistered