[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 406511170 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002185] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.005880] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006012] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007019] pid_max: default: 32768 minimum: 301 [ 0.008087] LSM: Security Framework initializing [ 0.009031] Yama: becoming mindful. [ 0.010021] SELinux: Initializing. [ 0.011043] *** VALIDATE selinux *** [ 0.018224] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022586] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023094] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024064] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025068] *** VALIDATE tmpfs *** [ 0.026346] *** VALIDATE proc *** [ 0.027167] *** VALIDATE cgroup *** [ 0.027859] *** VALIDATE cgroup2 *** [ 0.028197] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.029103] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.031021] Spectre V2 : User space: Vulnerable [ 0.032004] Speculative Store Bypass: Vulnerable [ 0.035684] debug: unmapping init [mem 0xffffffffba459000-0xffffffffba460fff] [ 0.037778] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.038465] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.039011] ... version: 2 [ 0.040023] ... bit width: 48 [ 0.040870] ... generic registers: 4 [ 0.041006] ... value mask: 0000ffffffffffff [ 0.042009] ... max period: 00007fffffffffff [ 0.043008] ... fixed-purpose events: 3 [ 0.043889] ... event mask: 000000070000000f [ 0.044285] rcu: Hierarchical SRCU implementation. [ 0.046320] smp: Bringing up secondary CPUs ... [ 0.047414] x86: Booting SMP configuration: [ 0.048024] .... node #0, CPUs: #1 #2 #3 [ 0.050592] smp: Brought up 1 node, 4 CPUs [ 0.051896] smpboot: Max logical packages: 1 [ 0.052006] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150609] node 0 deferred pages initialised in 97ms [ 0.154486] devtmpfs: initialized [ 0.155182] x86/mm: Memory block size: 128MB [ 0.157170] gcov: version magic: 0x41383552 [ 0.158658] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162055] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163284] pinctrl core: initialized pinctrl subsystem [ 0.165110] [ 0.165457] ************************************************************* [ 0.167008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168005] ** ** [ 0.169007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172008] ** ** [ 0.174008] ** This means that this kernel is built to expose internal ** [ 0.176006] ** IOMMU data structures, which may compromise security on ** [ 0.177005] ** your system. ** [ 0.178007] ** ** [ 0.180006] ** If you see this message and you are not debugging the ** [ 0.181005] ** kernel, report this immediately to your vendor! ** [ 0.183006] ** ** [ 0.184005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185005] ************************************************************* [ 0.187546] NET: Registered protocol family 16 [ 0.189431] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195029] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198033] cpuidle: using governor menu [ 0.199350] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.200444] PCI: Using configuration type 1 for base access [ 0.202122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209085] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.210011] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.212119] cryptd: max_cpu_qlen set to 1000 [ 0.214207] ACPI: Added _OSI(Module Device) [ 0.215009] ACPI: Added _OSI(Processor Device) [ 0.215891] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.217007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.220672] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.224421] ACPI: Interpreter enabled [ 0.225036] ACPI: PM: (supports S0 S3 S4 S5) [ 0.225940] ACPI: Using IOAPIC for interrupt routing [ 0.227049] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.229247] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.236500] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.238019] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.240013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.242049] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.245341] acpiphp: Slot [2] registered [ 0.247070] acpiphp: Slot [5] registered [ 0.247962] acpiphp: Slot [6] registered [ 0.248062] acpiphp: Slot [3] registered [ 0.250055] acpiphp: Slot [4] registered [ 0.250949] acpiphp: Slot [7] registered [ 0.251052] acpiphp: Slot [8] registered [ 0.251908] acpiphp: Slot [9] registered [ 0.253068] acpiphp: Slot [10] registered [ 0.253890] acpiphp: Slot [11] registered [ 0.255044] acpiphp: Slot [12] registered [ 0.255968] acpiphp: Slot [13] registered [ 0.256061] acpiphp: Slot [14] registered [ 0.258059] acpiphp: Slot [15] registered [ 0.259036] acpiphp: Slot [16] registered [ 0.260061] acpiphp: Slot [17] registered [ 0.261046] acpiphp: Slot [18] registered [ 0.262061] acpiphp: Slot [19] registered [ 0.264117] acpiphp: Slot [20] registered [ 0.265056] acpiphp: Slot [21] registered [ 0.265954] acpiphp: Slot [22] registered [ 0.267047] acpiphp: Slot [23] registered [ 0.268061] acpiphp: Slot [24] registered [ 0.268931] acpiphp: Slot [25] registered [ 0.270061] acpiphp: Slot [26] registered [ 0.271062] acpiphp: Slot [27] registered [ 0.271921] acpiphp: Slot [28] registered [ 0.274047] acpiphp: Slot [29] registered [ 0.274900] acpiphp: Slot [30] registered [ 0.276052] acpiphp: Slot [31] registered [ 0.277047] PCI host bridge to bus 0000:00 [ 0.278011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.280011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.282009] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.284011] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.286028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.287013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.288144] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.290755] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.292806] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.297358] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.299387] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.301008] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.302006] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.304007] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.305405] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.307507] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.309029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.310524] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.313006] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.319675] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.322008] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.327086] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.329882] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.333007] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.340009] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.345000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.348874] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.352008] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.358942] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.365503] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.367280] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.368182] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.370192] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.371126] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.374117] iommu: Default domain type: Passthrough [ 0.376284] SCSI subsystem initialized [ 0.377071] ACPI: bus type USB registered [ 0.377993] usbcore: registered new interface driver usbfs [ 0.378028] usbcore: registered new interface driver hub [ 0.379028] usbcore: registered new device driver usb [ 0.380072] pps_core: LinuxPPS API ver. 1 registered [ 0.381005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.383016] PTP clock support registered [ 0.384064] EDAC MC: Ver: 3.0.0 [ 0.385243] PCI: Using ACPI for IRQ routing [ 0.386552] NetLabel: Initializing [ 0.387006] NetLabel: domain hash size = 128 [ 0.387893] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.388077] NetLabel: unlabeled traffic allowed by default [ 0.389098] vgaarb: loaded [ 0.390209] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.392008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.396270] clocksource: Switched to clocksource kvm-clock [ 0.475014] VFS: Disk quotas dquot_6.6.0 [ 0.475996] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.477603] *** VALIDATE ramfs *** [ 0.478307] *** VALIDATE hugetlbfs *** [ 0.479335] pnp: PnP ACPI init [ 0.481049] pnp: PnP ACPI: found 6 devices [ 0.495349] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.497417] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.498765] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.500050] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.501625] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.503061] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.504811] NET: Registered protocol family 2 [ 0.506569] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.510112] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.512599] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.518159] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.522025] TCP: Hash tables configured (established 65536 bind 65536) [ 0.525519] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.528639] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.531570] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.534312] NET: Registered protocol family 1 [ 0.536064] RPC: Registered named UNIX socket transport module. [ 0.537402] RPC: Registered udp transport module. [ 0.538467] RPC: Registered tcp transport module. [ 0.539413] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.540904] NET: Registered protocol family 44 [ 0.541905] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.543082] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.544272] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.545600] PCI: CLS 0 bytes, default 64 [ 0.547135] Unpacking initramfs... [ 1.810798] debug: unmapping init [mem 0xffff8e9e3cc64000-0xffff8e9e3ffcffff] [ 1.813455] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.815083] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.817111] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.303329] Initialise system trusted keyrings [ 2.304444] Key type blacklist registered [ 2.305667] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.313567] zbud: loaded [ 2.315736] *** VALIDATE nfs *** [ 2.317017] *** VALIDATE nfs4 *** [ 2.318541] pstore: using deflate compression [ 2.321125] Platform Keyring initialized [ 2.393485] NET: Registered protocol family 38 [ 2.394728] Key type asymmetric registered [ 2.395984] Asymmetric key parser 'x509' registered [ 2.397045] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.398741] io scheduler mq-deadline registered [ 2.400171] io scheduler kyber registered [ 2.401396] io scheduler bfq registered [ 2.402839] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.405066] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.408122] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.410133] ACPI: Power Button [PWRF] [ 2.414174] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.419157] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.427023] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.452373] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.478905] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.482503] Non-volatile memory driver v1.3 [ 2.483665] Linux agpgart interface v0.103 [ 2.508107] virtio_blk virtio1: [vda] 134584 512-byte logical blocks (68.9 MB/65.7 MiB) [ 2.510224] vda: detected capacity change from 0 to 68907008 [ 2.522672] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.524751] vdb: detected capacity change from 0 to 1073741824 [ 2.529656] libphy: Fixed MDIO Bus: probed [ 2.533719] usbcore: registered new interface driver usbserial_generic [ 2.535703] usbserial: USB Serial support registered for generic [ 2.538337] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.541736] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.542842] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.544429] mousedev: PS/2 mouse device common for all mice [ 2.546666] rtc_cmos 00:05: RTC can wake from S4 [ 2.548938] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.549855] rtc_cmos 00:05: registered as rtc0 [ 2.553528] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.553700] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.557963] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.559585] intel_pstate: CPU model not supported [ 2.575753] hid: raw HID events driver (C) Jiri Kosina [ 2.577822] usbcore: registered new interface driver usbhid [ 2.579314] usbhid: USB HID core driver [ 2.580946] drop_monitor: Initializing network drop monitor service [ 2.582894] Initializing XFRM netlink socket [ 2.584347] NET: Registered protocol family 10 [ 2.587216] Segment Routing with IPv6 [ 2.588881] NET: Registered protocol family 17 [ 2.590871] mpls_gso: MPLS GSO support [ 2.595958] RAS: Correctable Errors collector initialized. [ 2.597914] AVX version of gcm_enc/dec engaged. [ 2.599396] AES CTR mode by8 optimization enabled [ 2.659370] sched_clock: Marking stable (2659344828, 0)->(3307526754, -648181926) [ 2.663325] registered taskstats version 1 [ 2.665321] Loading compiled-in X.509 certificates [ 2.668438] zswap: loaded using pool lzo/zbud [ 2.685263] Key type big_key registered [ 2.694134] Key type encrypted registered [ 2.695679] ima: No TPM chip found, activating TPM-bypass! [ 2.697622] ima: Allocated hash algorithm: sha1 [ 2.698834] ima: No architecture policies found [ 2.700088] evm: Initialising EVM extended attributes: [ 2.701938] evm: security.selinux [ 2.702728] evm: security.ima [ 2.703293] evm: security.capability [ 2.704113] evm: HMAC attrs: 0x1 [ 2.707185] rtc_cmos 00:05: setting system clock to 2026-03-16 13:34:31 UTC (1773668071) [ 2.712675] debug: unmapping init [mem 0xffffffffbb403000-0xffffffffbb5fffff] [ 2.715258] debug: unmapping init [mem 0xffffffffba182000-0xffffffffba458fff] [ 2.723093] Write protecting the kernel read-only data: 28672k [ 2.725861] debug: unmapping init [mem 0xffffffffb8803000-0xffffffffb89fffff] [ 2.728289] debug: unmapping init [mem 0xffffffffb9114000-0xffffffffb91fffff] [ 2.754923] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.760515] systemd[1]: Detected virtualization kvm. [ 2.761768] systemd[1]: Detected architecture x86-64. [ 2.762965] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.785579] systemd[1]: No hostname configured. [ 2.787070] systemd[1]: Set hostname to . [ 2.788549] random: systemd: uninitialized urandom read (16 bytes read) [ 2.790240] systemd[1]: Initializing machine ID from random generator. [ 2.826665] random: ln: uninitialized urandom read (6 bytes read) [ 2.887790] random: systemd: uninitialized urandom read (16 bytes read) [ 2.889886] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 2.893369] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 2.896593] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.394114] device-mapper: uevent: version 1.0.3 [ 3.396196] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [[ 4.053283] random: fast init done  OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.090872] virtio_net virtio0 ens2: renamed from eth0 [ 4.093631] scsi host0: ata_piix [ 4.096291] scsi host1: ata_piix [ 4.097726] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.099441] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 7.748061] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.217086] random: crng init done [ 9.218297] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.041837] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.298743] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.576570] SELinux: Disabled at runtime. [ 11.645381] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.654173] systemd[1]: Detected virtualization kvm. [ 11.656069] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.205180] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.207982] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.212528] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.217556] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.220568] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.229327] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.242903] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 12.272179] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.742106] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.142349] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.208831] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.261318] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.271230] EDAC sbridge: Ver: 1.1.2 [ 14.605089] Key type dns_resolver registered [ 14.956907] NFS: Registering the id_resolver key type [ 14.958255] Key type id_resolver registered [ 14.959464] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg459-client login: [ 41.984825] libcfs: loading out-of-tree module taints kernel. [ 42.037197] Key type ._llcrypt registered [ 42.038308] Key type .llcrypt registered [ 42.279190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 42.284310] alg: No test for adler32 (adler32-zlib) [ 43.344475] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 43.713254] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 45.336158] Key type lgssc registered [ 46.096427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 108.785028] Lustre: Mounted lustre-client [ 111.258928] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 121.131579] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing check_logdir /tmp/testlogs/ [ 122.815435] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing yml_node [ 124.558408] Lustre: DEBUG MARKER: Client: 2.17.51.1 [ 125.486678] Lustre: DEBUG MARKER: MDS: 2.17.51.1 [ 126.467295] Lustre: DEBUG MARKER: OSS: 2.17.51.1 [ 127.154371] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Mar 16 09:36:35 EDT 2026 [ 132.951140] LustreError: 7892:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e9e98faf800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 132.973070] Lustre: Unmounted lustre-client [ 144.626526] Key type lgssc unregistered [ 144.744523] LNet: 8339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 144.747424] LNetError: 8339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 144.756021] LNet: Removed LNI 192.168.204.59@tcp [ 145.017100] Key type .llcrypt unregistered [ 145.018105] Key type ._llcrypt unregistered [ 153.026593] Key type ._llcrypt registered [ 153.027425] Key type .llcrypt registered [ 153.097879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 153.103697] alg: No test for adler32 (adler32-zlib) [ 153.902699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 153.930274] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 153.940136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 155.190098] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 155.216399] LNet: 9208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 155.218944] LNetError: 9208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 155.227260] LNet: Removed LNI 192.168.204.59@tcp [ 155.452233] Key type .llcrypt unregistered [ 155.453336] Key type ._llcrypt unregistered [ 157.438355] Key type ._llcrypt registered [ 157.439326] Key type .llcrypt registered [ 157.889918] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:37:06 (1773668226) [ 157.998789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 158.004689] alg: No test for adler32 (adler32-zlib) [ 158.809512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 158.839508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 158.848734] LNet: 9625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 158.858831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 158.880055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 158.886596] LNet: 9632:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 161.154620] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:37:09 (1773668229) [ 161.181078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 161.191258] LNet: 10201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 161.200361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 161.231033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 163.308208] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:37:11 (1773668231) [ 163.344736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 163.353213] LNet: 10788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 163.362171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 163.393473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 165.456657] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:37:13 (1773668233) [ 165.480892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 165.489893] LNet: 11370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 165.498518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 165.530827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 167.579329] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:37:15 (1773668235) [ 167.607666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 167.626856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 167.661961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 167.806523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 169.928326] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:37:18 (1773668238) [ 169.952858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 169.961968] LNet: 12543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 169.964916] LNet: 12543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 169.973983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 169.998767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 170.119158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 170.253249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 172.254965] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:37:20 (1773668240) [ 172.283411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 172.300895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 172.332696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 174.505576] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:37:22 (1773668242) [ 174.533421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 174.552357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 174.581719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 174.600972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 174.778291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 174.794907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 174.952772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 174.972063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 175.128761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 175.153448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 175.309387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 175.326890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 175.494195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 175.512625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 175.666053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 175.679749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 175.833606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 175.846843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 176.000136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 176.014811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 176.154377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 176.168353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 176.303470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 176.315854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 176.452878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 176.467505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 176.605311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 176.619907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 178.888728] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:37:27 (1773668247) [ 178.937651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 178.953226] LNet: 14404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 178.957705] LNet: 14404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 178.968907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 179.007247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 179.022479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 181.463635] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:37:29 (1773668249) [ 181.496588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 181.513328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 181.546845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 181.563338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 183.727251] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:37:32 (1773668252) [ 183.792893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 183.806681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 183.831806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 183.845354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 186.100587] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:37:34 (1773668254) [ 186.132449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 186.158645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 186.192447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 186.207960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 188.504197] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:37:36 (1773668256) [ 188.533671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 188.551633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 188.583244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 188.599345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 190.796223] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:37:39 (1773668259) [ 190.824866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 190.845709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 190.882908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 190.899913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 193.301426] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:37:41 (1773668261) [ 193.332445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 193.350579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.386965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 193.530386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 195.897285] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:37:44 (1773668264) [ 195.926937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 195.935165] LNet: 18510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 195.938660] LNet: 18510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 195.950920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 196.034166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 196.121055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 198.614880] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:37:46 (1773668266) [ 198.665815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 198.686892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 198.750120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 198.848863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 201.264182] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:37:49 (1773668269) [ 201.299721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 201.321191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 201.540800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 201.639988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 203.984492] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:37:52 (1773668272) [ 204.024151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 204.045957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 204.082126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 204.104242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 206.518692] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:37:54 (1773668274) [ 206.555213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 206.571935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.603461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 206.622693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 209.108720] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:37:57 (1773668277) [ 209.172294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 209.189379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 209.214104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 211.714994] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:37:59 (1773668279) [ 211.757898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.785462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.822089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 214.143982] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:38:02 (1773668282) [ 214.198416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 214.220436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.252953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 214.406321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 216.876523] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:38:05 (1773668285) [ 216.920701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 216.940777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 216.977701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 216.991192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 219.486501] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:38:07 (1773668287) [ 219.537585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 219.559566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 219.592864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 219.616608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 222.122053] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:38:10 (1773668290) [ 222.186931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.207969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.248131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 222.265180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 224.846910] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:38:13 (1773668293) [ 224.903644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 224.920162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 224.949228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 224.964457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 227.406498] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:38:15 (1773668295) [ 227.440537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 227.460914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 227.495756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 227.524938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 227.534910] LNetError: 25711:0:(peer.c:2174:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 227.554343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 227.569928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 229.921251] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:38:18 (1773668298) [ 229.961508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.973653] LNet: 26282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 229.977245] LNet: 26282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 229.989611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.032908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8WChTrjkLy [ 230.069503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8WChTrjkLy [ 230.095979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8WChTrjkLy [ 232.687919] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:38:20 (1773668300) [ 232.729049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 232.749197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 232.788573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 232.807536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 232.829275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 235.560490] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 09:38:23 (1773668303) [ 235.597528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.616585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.654160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 236.759337] LNet: Message forwarding will be enabled in 2s [ 239.840218] LNet: Message forwarding enabled [ 239.998988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 240.019205] LNet: Message forwarding will stop in 2s to 6s [ 244.245289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 244.651574] LNet: Message forwarding disabled [ 245.273923] LNet: Message forwarding will be enabled in 2s [ 247.844095] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 09:38:36 (1773668316) [ 247.886995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 248.507948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 248.541097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 251.167406] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:38:39 (1773668319) [ 251.208923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 251.229993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 251.259622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 251.277539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 251.295334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 251.313879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 251.329746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 251.344287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 251.362885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 251.389210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 251.402601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 251.416523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 251.432492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 251.448018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 251.464442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 251.482435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 251.498623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 251.516202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 251.531556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 251.549565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 251.563744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 251.572233] LNetError: 28777:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 254.018849] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:38:42 (1773668322) [ 254.081139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 254.102722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 254.144546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 256.637539] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:38:44 (1773668324) [ 257.327826] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg459-server [ 258.105631] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:38:46 (1773668326) [ 258.140026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.164283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.202055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 258.213472] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 258.897556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[60-64]@tcp [ 258.907918] LNet: 30491:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 261.639470] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:38:49 (1773668329) [ 261.984233] LNetError: 30288:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.62@tcp has gone from up to down [ 262.481815] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg459-server [ 263.157357] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:38:51 (1773668331) [ 263.195288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 265.248151] LNetError: 30280:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.64@tcp has gone from up to down [ 265.252352] LNetError: 30280:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 265.255463] LNetError: 31420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 265.265638] LNet: Removed LNI 192.168.204.59@tcp [ 265.273674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.295146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 265.307929] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 265.581063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[60-65/2]@tcp [ 265.591787] LNet: 31543:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 265.596288] LNet: 31543:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 265.605980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[60-65/2]@tcp [ 268.166485] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:38:56 (1773668336) [ 268.223839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.234125] LNetError: 32135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 268.240810] LNet: Removed LNI 192.168.204.59@tcp [ 268.251414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.284676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 268.308936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.325464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.353821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 268.377320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.389504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.409220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 268.434519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.450990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.481475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 268.515705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.535147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.561153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 268.595675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.610603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.639219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 268.670615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.685412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.704874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 268.730399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.750806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.777773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 268.811093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.825074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.844719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 268.871991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.885254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.909528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 268.940488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.956929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.980862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 269.007975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 269.025119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 269.052590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 271.193932] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:38:59 (1773668339) [ 271.827918] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg459-server [ 272.406202] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:39:00 (1773668340) [ 273.058946] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg459-server [ 273.690382] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:39:01 (1773668341) [ 273.997558] Key type .llcrypt unregistered [ 273.998883] Key type ._llcrypt unregistered [ 276.023807] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 276.077548] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 276.079417] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 276.082627] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 278.268894] Key type ._llcrypt registered [ 278.269977] Key type .llcrypt registered [ 278.335148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 278.341743] alg: No test for adler32 (adler32-zlib) [ 279.147231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.168351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 279.177663] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 279.409763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 279.418635] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 280.070758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 280.080736] LNet: Removed LNI 10.1.2.3@tcp [ 280.123668] LNet: 34254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 280.126356] LNetError: 34254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 280.133322] LNet: Removed LNI 192.168.204.59@tcp [ 280.350361] Key type .llcrypt unregistered [ 280.351910] Key type ._llcrypt unregistered [ 283.841722] Key type ._llcrypt registered [ 283.843139] Key type .llcrypt registered [ 284.339701] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:39:12 (1773668352) [ 284.497410] Key type .llcrypt unregistered [ 284.498944] Key type ._llcrypt unregistered [ 286.408864] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 286.467323] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 286.469345] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 286.470955] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 288.652703] Key type ._llcrypt registered [ 288.653849] Key type .llcrypt registered [ 288.712510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 288.718155] alg: No test for adler32 (adler32-zlib) [ 289.517930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.540255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 289.548380] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 289.788110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 289.796735] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 290.640549] LNet: 35745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.643208] LNetError: 35745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 290.645836] LNet: Removed LNI 192.168.204.59@tcp [ 290.656359] LNet: Removed LNI 10.1.2.3@tcp [ 290.889504] Key type .llcrypt unregistered [ 290.891313] Key type ._llcrypt unregistered [ 294.339556] Key type ._llcrypt registered [ 294.341039] Key type .llcrypt registered [ 294.785445] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:39:23 (1773668363) [ 294.947433] Key type .llcrypt unregistered [ 294.948602] Key type ._llcrypt unregistered [ 296.902532] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 296.966537] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 296.971401] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 296.973699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 299.185063] Key type ._llcrypt registered [ 299.186329] Key type .llcrypt registered [ 299.259430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 299.265744] alg: No test for adler32 (adler32-zlib) [ 300.083716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.107651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 300.119481] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 300.961294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 300.975760] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 301.795970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 301.805257] LNet: Removed LNI 10.1.2.3@tcp [ 301.811755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 301.828654] LNet: Removed LNI 10.1.2.31@tcp [ 301.836109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 301.845190] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 302.106517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 302.115815] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 302.426986] LNet: 37662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.429430] LNetError: 37662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 302.432161] LNet: Removed LNI 10.1.2.3@tcp [ 302.438359] LNet: Removed LNI 10.1.2.31@tcp [ 302.678067] Key type .llcrypt unregistered [ 302.681081] Key type ._llcrypt unregistered [ 306.508097] Key type ._llcrypt registered [ 306.509196] Key type .llcrypt registered [ 307.009572] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:39:35 (1773668375) [ 307.143522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 307.149499] alg: No test for adler32 (adler32-zlib) [ 307.964094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.988679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 308.000921] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 308.333549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.342473] LNet: 38613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 308.345296] LNetError: 38613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 308.353475] LNet: Removed LNI 192.168.204.59@tcp [ 308.361929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.386912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.59@tcp [ 308.400040] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 308.725143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.59@tcp [ 308.743360] LNet: Removed LNI 192.168.204.59@tcp [ 309.036386] Key type .llcrypt unregistered [ 309.039670] Key type ._llcrypt unregistered [ 312.932643] Key type ._llcrypt registered [ 312.933989] Key type .llcrypt registered [ 313.410326] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:39:41 (1773668381) [ 314.028029] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg459-server [ 314.642265] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:39:42 (1773668382) [ 314.816248] Key type .llcrypt unregistered [ 314.818334] Key type ._llcrypt unregistered [ 316.787336] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 316.836367] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 316.839938] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 316.842685] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 319.054299] Key type ._llcrypt registered [ 319.055529] Key type .llcrypt registered [ 319.124256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 319.131347] alg: No test for adler32 (adler32-zlib) [ 319.939232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.964640] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 320.246444] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 321.201631] LNet: 40582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.205541] LNetError: 40582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 321.208965] LNet: Removed LNI 192.168.204.59@tcp1 [ 321.217521] LNet: Removed LNI 10.1.2.3@tcp2 [ 321.497735] Key type .llcrypt unregistered [ 321.500690] Key type ._llcrypt unregistered [ 325.761243] Key type ._llcrypt registered [ 325.762544] Key type .llcrypt registered [ 326.329889] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:39:54 (1773668394) [ 326.485994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 326.491458] alg: No test for adler32 (adler32-zlib) [ 327.299336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.321408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 327.332866] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 327.687537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.696140] LNet: 41549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.699436] LNetError: 41549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 327.711439] LNet: Removed LNI 192.168.204.59@tcp [ 327.721174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.753832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 327.769612] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 330.453251] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:39:58 (1773668398) [ 330.497571] LNet: 42134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.501596] LNetError: 42134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 330.510995] LNet: Removed LNI 192.168.204.59@tcp [ 330.780677] Key type .llcrypt unregistered [ 330.782204] Key type ._llcrypt unregistered [ 332.818525] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 332.894153] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 332.900565] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 332.902861] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 335.095240] Key type ._llcrypt registered [ 335.096869] Key type .llcrypt registered [ 335.177592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 335.187146] alg: No test for adler32 (adler32-zlib) [ 335.995959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.019342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 336.031361] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 336.323918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 336.332771] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 337.208329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.217501] LNet: 42882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.219560] LNetError: 42882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 337.221634] LNet: Removed LNI 192.168.204.59@tcp [ 337.234444] LNet: Removed LNI 10.1.2.3@tcp [ 337.244105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.283527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 337.297512] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 337.300395] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 337.502114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.511306] LNet: Removed LNI 192.168.204.59@tcp [ 337.520806] LNet: Removed LNI 10.1.2.3@tcp [ 337.532503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.566925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 337.589648] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 337.594322] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 337.796619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 337.813697] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 338.315759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.328073] LNet: 43072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.330413] LNet: 43072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 338.335726] LNetError: 43072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 338.338637] LNetError: 43072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 338.342576] LNet: Removed LNI 192.168.204.59@tcp [ 338.344719] LNet: Removed LNI 10.1.2.3@tcp [ 338.354409] LNet: Removed LNI 192.168.204.59@tcp2 [ 338.366407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.403879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 338.425282] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 338.427921] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 338.431975] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 341.391254] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:40:09 (1773668409) [ 341.457164] LNet: 43669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.459478] LNetError: 43669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 341.463057] LNetError: 43669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 341.465728] LNet: Removed LNI 192.168.204.59@tcp [ 341.467937] LNet: Removed LNI 10.1.2.3@tcp [ 341.474151] LNet: Removed LNI 192.168.204.59@tcp2 [ 341.782626] Key type .llcrypt unregistered [ 341.784302] Key type ._llcrypt unregistered [ 344.235872] Key type ._llcrypt registered [ 344.237345] Key type .llcrypt registered [ 344.318601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 344.325470] alg: No test for adler32 (adler32-zlib) [ 345.156314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.184226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 345.197614] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 345.560211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.571921] LNet: 44064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.574490] LNetError: 44064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 345.585149] LNet: Removed LNI 192.168.204.59@tcp [ 345.595616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.634599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 345.648391] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 345.837506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.858075] LNet: Removed LNI 192.168.204.59@tcp [ 345.866640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.892911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 345.905432] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 346.075250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.085188] LNet: 44124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.088730] LNet: 44124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 346.093416] LNetError: 44124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 346.096301] LNetError: 44124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 346.105568] LNet: Removed LNI 192.168.204.59@tcp [ 346.114797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.146419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 346.159179] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 346.363511] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 346.442929] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 346.445602] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 346.447995] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 348.601807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.611206] LNet: 44201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.614318] LNetError: 44201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 348.622479] LNet: Removed LNI 192.168.204.59@tcp [ 348.631969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.661251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 348.672816] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 351.422236] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:40:19 (1773668419) [ 351.478765] LNet: 44794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.483253] LNetError: 44794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 351.494422] LNet: Removed LNI 192.168.204.59@tcp [ 351.757118] Key type .llcrypt unregistered [ 351.759556] Key type ._llcrypt unregistered [ 354.115126] Key type ._llcrypt registered [ 354.116188] Key type .llcrypt registered [ 354.220232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 354.227252] alg: No test for adler32 (adler32-zlib) [ 355.043868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.073264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 355.091169] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 355.481914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.491519] LNet: 45196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.494052] LNetError: 45196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 355.505366] LNet: Removed LNI 192.168.204.59@tcp [ 355.515773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.548331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 355.562799] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 358.307196] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:40:26 (1773668426) [ 358.353631] LNet: 45781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.361657] LNetError: 45781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 358.373225] LNet: Removed LNI 192.168.204.59@tcp [ 358.626176] Key type .llcrypt unregistered [ 358.627399] Key type ._llcrypt unregistered [ 360.700872] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 360.742183] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 360.744086] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 360.746198] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 362.997472] Key type ._llcrypt registered [ 362.998886] Key type .llcrypt registered [ 363.072805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 363.079632] alg: No test for adler32 (adler32-zlib) [ 363.886059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.922445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 363.934198] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 364.333590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.344358] LNet: 46236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.354262] LNetError: 46236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 364.368333] LNet: Removed LNI 192.168.204.59@tcp [ 364.382541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.413233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 364.426937] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 367.389839] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:40:35 (1773668435) [ 367.446774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.463490] LNet: 46831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.467195] LNetError: 46831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 367.478685] LNet: Removed LNI 192.168.204.59@tcp [ 367.490770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.538124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 370.147544] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:40:38 (1773668438) [ 370.175114] LNet: 47409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.423782] Key type .llcrypt unregistered [ 370.425048] Key type ._llcrypt unregistered [ 372.836882] Key type ._llcrypt registered [ 372.838263] Key type .llcrypt registered [ 372.942072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 372.949429] alg: No test for adler32 (adler32-zlib) [ 373.770543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.797749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 373.808901] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 374.188099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.198440] LNet: 47813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.202167] LNetError: 47813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 374.212807] LNet: Removed LNI 192.168.204.59@tcp [ 374.223730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.254934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 374.268035] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 376.986034] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 09:40:45 (1773668445) [ 377.044824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.058432] LNet: 48406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.062279] LNetError: 48406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 377.069476] LNet: Removed LNI 192.168.204.59@tcp [ 377.080214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.113978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 379.739226] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:40:47 (1773668447) [ 380.387140] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 381.114787] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:40:49 (1773668449) [ 381.740338] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 382.505145] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:40:50 (1773668450) [ 383.132370] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 383.863742] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:40:52 (1773668452) [ 384.528567] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 385.224631] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:40:53 (1773668453) [ 385.933031] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 386.709279] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:40:54 (1773668454) [ 386.751541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.764276] LNet: 50451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.778078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.824198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 386.838464] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 387.177889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.59 [ 387.187941] LNet: 50589:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 387.192760] LNet: Removed LNI 192.168.204.59@tcp [ 389.843059] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:40:57 (1773668457) [ 389.872651] LNet: 51152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.875910] LNetError: 51152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 389.883681] LNet: Removed LNI 192.168.204.59@tcp [ 390.140920] Key type .llcrypt unregistered [ 390.142799] Key type ._llcrypt unregistered [ 392.395794] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 392.517623] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 392.520263] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 392.522660] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 394.856104] Key type ._llcrypt registered [ 394.857389] Key type .llcrypt registered [ 394.935975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 394.942326] alg: No test for adler32 (adler32-zlib) [ 395.947377] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 398.507082] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:41:06 (1773668466) [ 398.541390] LNet: 52055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.544455] LNetError: 52055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 398.555792] LNet: Removed LNI 10.1.2.3@tcp [ 398.835605] Key type .llcrypt unregistered [ 398.837321] Key type ._llcrypt unregistered [ 400.989529] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 401.111962] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 401.114256] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 401.116956] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 403.442273] Key type ._llcrypt registered [ 403.443499] Key type .llcrypt registered [ 403.518323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 403.524031] alg: No test for adler32 (adler32-zlib) [ 404.368477] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 406.849433] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:41:15 (1773668475) [ 406.897086] LNet: 52947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 406.900689] LNetError: 52947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 406.910227] LNet: Removed LNI 10.1.2.3@tcp [ 407.177639] Key type .llcrypt unregistered [ 407.179442] Key type ._llcrypt unregistered [ 409.586819] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 409.704133] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 409.708975] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 409.711411] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 412.051435] Key type ._llcrypt registered [ 412.052839] Key type .llcrypt registered [ 412.112224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 412.120786] alg: No test for adler32 (adler32-zlib) [ 412.960055] LNet: 53255:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 412.966512] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 415.897194] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:41:23 (1773668483) [ 415.933853] LNet: 53839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 415.937903] LNetError: 53839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 415.946715] LNet: Removed LNI 10.1.2.3@tcp [ 416.223650] Key type .llcrypt unregistered [ 416.227311] Key type ._llcrypt unregistered [ 418.681282] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 418.794417] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 418.797116] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 418.799502] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 421.167600] Key type ._llcrypt registered [ 421.168838] Key type .llcrypt registered [ 421.263374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 421.271542] alg: No test for adler32 (adler32-zlib) [ 422.083444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.119933] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 425.341218] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:41:33 (1773668493) [ 425.513660] LNet: 55073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.517581] LNetError: 55073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 425.529662] LNet: Removed LNI 10.1.2.3@tcp [ 425.812562] Key type .llcrypt unregistered [ 425.814182] Key type ._llcrypt unregistered [ 428.192624] Key type ._llcrypt registered [ 428.197405] Key type .llcrypt registered [ 428.291674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 428.299665] alg: No test for adler32 (adler32-zlib) [ 429.128029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 429.142815] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 432.539983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 432.814990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_interrupt -r 1 [ 432.836442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_interrupt -r 1 [ 432.856509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 432.871644] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 432.964809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 432.982367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 432.997288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 433.011813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 433.038111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 433.050948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 433.065934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 433.080064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 433.089178] LNet: 55510:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 433.099034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 433.116050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 433.132271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 433.155533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 433.171440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 433.189786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 433.205492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 433.422635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_dropped -r 1 [ 433.443232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_dropped -r 1 [ 433.460880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 433.473032] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 433.591037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 433.606452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 433.626398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 433.638885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 433.652990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 433.667968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 433.683765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 433.700461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 433.709590] LNet: 55619:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 433.718330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 433.731834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 433.749602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 433.769803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 433.783831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 433.804155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 433.823192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 434.058359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_aborted -r 1 [ 434.076130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_aborted -r 1 [ 434.090070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 434.186881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 434.204420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 434.220964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 434.234161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 434.248969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 434.263241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 434.272284] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 434.275797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 434.289789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 434.306711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 434.321856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 434.338088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 434.356797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 434.370555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 434.385399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 434.403149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 434.642754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_no_route -r 1 [ 434.665782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_no_route -r 1 [ 434.686773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 434.698851] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 434.702243] LNet: Skipped 2 previous similar messages [ 434.805735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 434.824569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 434.841765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 434.859621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 434.877303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 434.896831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 434.915746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 434.932809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 434.942892] LNet: 55837:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 434.946734] LNet: 55837:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 434.957452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 434.975761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 434.992642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 435.010327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 435.028521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 435.048175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 435.064880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 435.306422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_timeout -r 1 [ 435.332787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_timeout -r 1 [ 435.352555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 435.486229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 435.510496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 435.532121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 435.552582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 435.569459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 435.590581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 435.605089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 435.622777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 435.636896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 435.656653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 435.670871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 435.686123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 435.700640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 435.715937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 435.730082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 435.931847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_error -r 1 [ 435.953098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_error -r 1 [ 435.975827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 436.080621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 436.095625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 436.117886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 436.138274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 436.152304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 436.167584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 436.180792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 436.197428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 436.212091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 436.228635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 436.244229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 436.258625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 436.273647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 436.294483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 436.311645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 436.412755] LNet: 56096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 436.417428] LNetError: 56096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 436.433359] LNet: Removed LNI 192.168.204.59@tcp [ 436.582524] Key type .llcrypt unregistered [ 436.584174] Key type ._llcrypt unregistered [ 442.427073] Key type ._llcrypt registered [ 442.428590] Key type .llcrypt registered [ 442.954163] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:41:51 (1773668511) [ 443.320369] Key type .llcrypt unregistered [ 443.321701] Key type ._llcrypt unregistered [ 445.416168] Key type ._llcrypt registered [ 445.417609] Key type .llcrypt registered [ 445.488720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 445.497057] alg: No test for adler32 (adler32-zlib) [ 446.305747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 446.318331] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 449.436893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 453.027818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.043946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 453.054148] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 453.647524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_interrupt -r 1 [ 453.665324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_interrupt -r 1 [ 453.684931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_interrupt -r 1 [ 453.705642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_interrupt -r 1 [ 453.731366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_interrupt -r 1 [ 453.764659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_interrupt -r 1 [ 453.788611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_interrupt -r 1 [ 453.805505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_interrupt -r 1 [ 453.823966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 453.834053] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 453.836929] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 453.917259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 453.932525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 453.950080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 453.964263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 453.984503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 454.004599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 454.022379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 454.038950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 454.047576] LNet: 57559:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 454.056627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 454.071375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 454.086031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 454.104183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 454.119136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 454.134149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 454.154059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 454.383373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_dropped -r 1 [ 454.401842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_dropped -r 1 [ 454.419196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_dropped -r 1 [ 454.434349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_dropped -r 1 [ 454.451600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_dropped -r 1 [ 454.467678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_dropped -r 1 [ 454.484653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_dropped -r 1 [ 454.507062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_dropped -r 1 [ 454.526666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 454.536085] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 454.538706] LNet: Skipped 2 previous similar messages [ 454.639092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 454.654537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 454.672276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 454.692540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 454.708316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 454.722340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 454.739456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 454.754816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 454.764076] LNet: 57680:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 454.774163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 454.789534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 454.802905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 454.819601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 454.837551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 454.854362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 454.875705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 455.077183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_aborted -r 1 [ 455.092302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_aborted -r 1 [ 455.113453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_aborted -r 1 [ 455.129060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_aborted -r 1 [ 455.141215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_aborted -r 1 [ 455.155840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_aborted -r 1 [ 455.172570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_aborted -r 1 [ 455.190937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_aborted -r 1 [ 455.219624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 455.334935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 455.349137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 455.363805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 455.378207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 455.394183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 455.405926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 455.417363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 455.428599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 455.441071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 455.451973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 455.469370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 455.484869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 455.501074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 455.514337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 455.529154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 455.742895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_no_route -r 1 [ 455.760938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_no_route -r 1 [ 455.777727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_no_route -r 1 [ 455.796838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_no_route -r 1 [ 455.814206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_no_route -r 1 [ 455.831793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_no_route -r 1 [ 455.850111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_no_route -r 1 [ 455.867428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_no_route -r 1 [ 455.883460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 455.892556] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 455.895819] LNet: Skipped 7 previous similar messages [ 455.991265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 456.006642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 456.024818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 456.040565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 456.057730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 456.075383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 456.096267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 456.117371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 456.130458] LNet: 57922:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 456.136225] LNet: 57922:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 456.145297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 456.158153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 456.173699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 456.197109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 456.212472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 456.231810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 456.247143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 456.454950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_timeout -r 1 [ 456.474465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_timeout -r 1 [ 456.491583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_timeout -r 1 [ 456.510444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_timeout -r 1 [ 456.527694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_timeout -r 1 [ 456.545958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_timeout -r 1 [ 456.562888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_timeout -r 1 [ 456.582889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_timeout -r 1 [ 456.600276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 456.711508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 456.723857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 456.738138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 456.752240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 456.769844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 456.783845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 456.796560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 456.810437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 456.823554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 456.836586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 456.849852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 456.863877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 456.881100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 456.897203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 456.911802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 457.102491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_error -r 1 [ 457.119442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_error -r 1 [ 457.138360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_error -r 1 [ 457.156873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_error -r 1 [ 457.173210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_error -r 1 [ 457.192078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_error -r 1 [ 457.210838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_error -r 1 [ 457.228565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_error -r 1 [ 457.241086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 457.322991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 457.339940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 457.353712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 457.368885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 457.380214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 457.392184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 457.405200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 457.418134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 457.432640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 457.446803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 457.462354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 457.477113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 457.492308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 457.506824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 457.523119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 459.801402] LNet: 58240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 459.804734] LNetError: 58240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 459.807941] LNet: Removed LNI 192.168.204.59@tcp [ 460.838413] LNet: Removed LNI 192.168.204.59@tcp1 [ 461.020539] Key type .llcrypt unregistered [ 461.021459] Key type ._llcrypt unregistered [ 466.750757] Key type ._llcrypt registered [ 466.752413] Key type .llcrypt registered [ 467.366083] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:42:15 (1773668535) [ 467.721599] Key type .llcrypt unregistered [ 467.723580] Key type ._llcrypt unregistered [ 469.958227] Key type ._llcrypt registered [ 469.959706] Key type .llcrypt registered [ 470.041167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 470.052256] alg: No test for adler32 (adler32-zlib) [ 470.898673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 470.915652] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 474.715285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 475.206635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e remote_dropped -r 1 [ 475.234245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e remote_dropped -r 1 [ 475.255508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 475.265429] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 475.449592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 475.472494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 475.500849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 475.530370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 475.554725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 475.587567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 475.619851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 475.677313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 475.694293] LNet: 59486:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 475.707179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 475.732598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 475.760932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 475.794767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 475.824367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 475.881103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 475.917498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 476.341265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e remote_error -r 1 [ 476.376422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e remote_error -r 1 [ 476.403639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 476.414182] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 476.615506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 476.640853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 476.668761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 476.693207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 476.716243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 476.741830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 476.764588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 476.786869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 476.797934] LNet: 59595:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 476.810773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 476.847952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 476.879218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 476.904913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 476.942915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 476.977669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 477.007535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 477.673299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e remote_timeout -r 1 [ 477.712659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e remote_timeout -r 1 [ 477.745593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 477.777925] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 478.001450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 478.026145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 478.045186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 478.068780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 478.090635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 478.110866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 478.161798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 478.180883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 478.193251] LNet: 59704:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 478.213919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 478.240522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 478.278110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 478.317686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 478.381130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 478.427733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 478.467363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 478.805760] LNet: 59746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 478.812993] LNetError: 59746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 478.828738] LNet: Removed LNI 192.168.204.59@tcp [ 479.190832] Key type .llcrypt unregistered [ 479.193163] Key type ._llcrypt unregistered [ 495.305946] Key type ._llcrypt registered [ 495.308241] Key type .llcrypt registered [ 496.325856] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:42:43 (1773668563) [ 497.006301] Key type .llcrypt unregistered [ 497.008395] Key type ._llcrypt unregistered [ 502.746168] Key type ._llcrypt registered [ 502.749723] Key type .llcrypt registered [ 503.059504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 503.095359] alg: No test for adler32 (adler32-zlib) [ 503.992965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 504.014153] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 509.679006] hrtimer: interrupt took 2997091 ns [ 511.270233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 520.091698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.147823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 520.174213] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 521.336291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e remote_dropped -r 1 [ 521.366560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e remote_dropped -r 1 [ 521.402592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e remote_dropped -r 1 [ 521.428967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e remote_dropped -r 1 [ 521.458758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e remote_dropped -r 1 [ 521.489772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e remote_dropped -r 1 [ 521.531402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e remote_dropped -r 1 [ 521.585482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e remote_dropped -r 1 [ 521.628907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 521.655853] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 521.660916] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.159@tcp [ 521.898453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 521.923667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 521.948866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 521.973814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 522.000774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 522.022644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 522.043432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 522.064866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 522.074854] LNet: 61212:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 522.089211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 522.117805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 522.143646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 522.173279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 522.206815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 522.240210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 522.260350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 522.296229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 522.763178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e remote_error -r 1 [ 522.790961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e remote_error -r 1 [ 522.818908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e remote_error -r 1 [ 522.847723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e remote_error -r 1 [ 522.877493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e remote_error -r 1 [ 522.905666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e remote_error -r 1 [ 522.955854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e remote_error -r 1 [ 523.012842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e remote_error -r 1 [ 523.048928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 523.063419] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 523.066207] LNet: Skipped 2 previous similar messages [ 523.260509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 523.280711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 523.302716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 523.324299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 523.344828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 523.372916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 523.393379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 523.417219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 523.429164] LNet: 61335:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 523.443953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 523.467517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 523.489385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 523.509730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 523.541371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 523.578517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 523.608778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 523.641925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 524.062457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e remote_timeout -r 1 [ 524.087923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e remote_timeout -r 1 [ 524.114888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e remote_timeout -r 1 [ 524.145875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e remote_timeout -r 1 [ 524.171284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e remote_timeout -r 1 [ 524.195249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e remote_timeout -r 1 [ 524.221898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e remote_timeout -r 1 [ 524.245745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e remote_timeout -r 1 [ 524.269702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 524.282320] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 524.448130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 524.472301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 524.494928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 524.515817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 524.540252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 524.564855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 524.589486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 524.614373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 524.626359] LNet: 61458:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 524.638917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 524.660964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 524.685198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 524.708082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 524.728766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 524.751744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 524.779442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 524.803385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 528.325953] LNet: 61536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.332284] LNetError: 61536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 528.339903] LNet: Removed LNI 192.168.204.59@tcp [ 528.347210] LNet: Removed LNI 192.168.204.59@tcp1 [ 528.589742] Key type .llcrypt unregistered [ 528.592467] Key type ._llcrypt unregistered [ 536.967634] Key type ._llcrypt registered [ 536.969206] Key type .llcrypt registered [ 537.751324] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:43:25 (1773668605) [ 538.029475] Key type .llcrypt unregistered [ 538.032224] Key type ._llcrypt unregistered [ 540.635467] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 540.700498] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 540.705372] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 540.710081] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 543.194205] Key type ._llcrypt registered [ 543.195888] Key type .llcrypt registered [ 543.303283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 543.313067] alg: No test for adler32 (adler32-zlib) [ 544.176479] LNet: 62613:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 544.187894] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 544.207133] LNet: 62627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 544.210324] LNetError: 62627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 544.221730] LNet: Removed LNI 192.168.204.59@tcp [ 544.656502] Key type .llcrypt unregistered [ 544.658861] Key type ._llcrypt unregistered [ 548.370922] Key type ._llcrypt registered [ 548.372530] Key type .llcrypt registered [ 548.494266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 548.503613] alg: No test for adler32 (adler32-zlib) [ 549.350337] LNet: 62886:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 549.364915] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 549.366936] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 549.402885] LNet: 62900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 549.407947] LNetError: 62900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 549.410659] LNet: Removed LNI 192.168.204.59@tcp [ 549.427881] LNet: Removed LNI 10.1.2.3@tcp1 [ 549.800428] Key type .llcrypt unregistered [ 549.801740] Key type ._llcrypt unregistered [ 553.416872] Key type ._llcrypt registered [ 553.418777] Key type .llcrypt registered [ 553.502386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 553.510740] alg: No test for adler32 (adler32-zlib) [ 554.369379] LNet: 63159:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 554.378837] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 554.381722] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 554.402125] LNet: 63173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 554.405570] LNetError: 63173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 554.409721] LNet: Removed LNI 192.168.204.59@tcp [ 554.426192] LNet: Removed LNI 10.1.2.3@tcp [ 554.813850] Key type .llcrypt unregistered [ 554.815627] Key type ._llcrypt unregistered [ 558.178169] Key type ._llcrypt registered [ 558.179845] Key type .llcrypt registered [ 558.322717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 558.329527] alg: No test for adler32 (adler32-zlib) [ 559.185855] LNet: 63433:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 559.199675] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 559.201799] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 559.225839] LNet: 63447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 559.230468] LNetError: 63447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 559.237132] LNet: Removed LNI 192.168.204.59@tcp [ 559.253203] LNet: Removed LNI 10.1.2.3@tcp [ 559.694367] Key type .llcrypt unregistered [ 559.697088] Key type ._llcrypt unregistered [ 563.205753] Key type ._llcrypt registered [ 563.208337] Key type .llcrypt registered [ 563.337861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 563.347723] alg: No test for adler32 (adler32-zlib) [ 564.213223] LNet: 63711:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 564.221088] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 564.223284] LNetError: 63711:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 564.227308] LNetError: Error -100 starting up LNI tcp [ 564.238941] LNet: Removed LNI 192.168.204.59@tcp [ 564.616424] Key type .llcrypt unregistered [ 564.617883] Key type ._llcrypt unregistered [ 570.961494] Key type ._llcrypt registered [ 570.963037] Key type .llcrypt registered [ 571.817749] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:43:59 (1773668639) [ 572.410271] Key type .llcrypt unregistered [ 572.411863] Key type ._llcrypt unregistered [ 576.024742] Key type ._llcrypt registered [ 576.026611] Key type .llcrypt registered [ 576.138878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 576.148424] alg: No test for adler32 (adler32-zlib) [ 577.003896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 577.026329] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 582.010597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 582.433736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e network_timeout -r 1 [ 582.455969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e network_timeout -r 1 [ 582.476940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 582.489297] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 582.596721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 582.615864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 582.634604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 582.657811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 582.678286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 582.696695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 582.714806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 582.737264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 582.751355] LNet: 64965:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 582.765830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 582.786662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 582.811716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 582.830569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 582.854257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 582.873759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 583.028922] LNet: 65004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 583.031960] LNetError: 65004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 583.048407] LNet: Removed LNI 192.168.204.59@tcp [ 583.302847] Key type .llcrypt unregistered [ 583.304900] Key type ._llcrypt unregistered [ 594.597737] Key type ._llcrypt registered [ 594.599886] Key type .llcrypt registered [ 594.756372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 594.777677] alg: No test for adler32 (adler32-zlib) [ 595.647609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 595.672758] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 600.330028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 604.725810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.750477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 604.773427] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 605.595514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e network_timeout -r 1 [ 605.615828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e network_timeout -r 1 [ 605.640910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e network_timeout -r 1 [ 605.665374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e network_timeout -r 1 [ 605.681368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e network_timeout -r 1 [ 605.698142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e network_timeout -r 1 [ 605.715763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e network_timeout -r 1 [ 605.734473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e network_timeout -r 1 [ 605.763966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 605.775371] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 605.897645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 605.920226] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 605.922123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 605.923106] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.159@tcp [ 605.946730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 605.974496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 606.002492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 606.025648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 606.046732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 606.066284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 606.082496] LNet: 65881:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 606.101592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 606.122624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 606.137139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 606.162252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 606.183075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 606.207426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 608.860714] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 609.017555] LNet: 65955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.021369] LNetError: 65955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 609.027330] LNet: Removed LNI 192.168.204.59@tcp [ 609.038534] LNet: Removed LNI 192.168.204.59@tcp1 [ 609.232507] Key type .llcrypt unregistered [ 609.236199] Key type ._llcrypt unregistered [ 616.991415] Key type ._llcrypt registered [ 616.994565] Key type .llcrypt registered [ 617.724686] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:44:45 (1773668685) [ 617.932431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 617.944550] alg: No test for adler32 (adler32-zlib) [ 618.780153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.807159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 618.826690] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 619.211970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 619.227592] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 619.718499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 619.764952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 619.793715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 619.824345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 619.847862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 619.865774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 619.893561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.59@tcp [ 619.910143] LNetError: 67052:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.59@tcp to recovery [ 620.834948] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 620.836583] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 622.880631] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 623.206664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 626.977616] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 627.541993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 627.563267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 627.611895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.628165] LNet: 67169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.632863] LNetError: 67169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 627.636201] LNet: Removed LNI 192.168.204.59@tcp [ 627.649736] LNet: Removed LNI 192.168.204.59@tcp1 [ 627.663423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.690480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 627.705205] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 628.103034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 628.116855] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 628.568688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 628.591892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 628.624760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 628.649321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 628.679940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 628.703546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 628.729605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 628.758294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.59@tcp [ 628.772379] LNetError: 67442:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.59@tcp to recovery [ 629.730568] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 632.058746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 635.873694] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 635.879324] LNet: Skipped 1 previous similar message [ 640.606253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 640.631737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 640.652497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 644.279539] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:45:12 (1773668712) [ 644.326974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.348352] LNet: 68162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 644.352512] LNetError: 68162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 644.357848] LNetError: 68162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 644.362602] LNet: Removed LNI 192.168.204.59@tcp [ 644.371663] LNet: Removed LNI 192.168.204.59@tcp1 [ 644.397216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.453796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 644.467755] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 644.901450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 644.913508] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 645.445849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 645.496923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 645.520638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 645.549613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 645.578068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 645.603834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 645.630932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.59@tcp --health 0 [ 645.696484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 646.496800] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.59@tcp [ 646.497363] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 646.507293] LNet: Skipped 1 previous similar message [ 661.538413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 661.637202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 661.660651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 661.685955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.59@tcp --health 500 [ 661.765057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 668.107733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 668.152561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.164143] LNet: 68667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.170641] LNetError: 68667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.173969] LNetError: 68667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 668.177542] LNet: Removed LNI 192.168.204.59@tcp [ 668.190181] LNet: Removed LNI 192.168.204.59@tcp1 [ 668.198914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.238896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 668.254112] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 668.662888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 668.679144] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 669.190616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 669.220193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 669.252665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 669.271920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 669.301301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 669.326729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 669.358243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 669.388690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.59@tcp --health 0 [ 670.240912] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 670.245697] LNet: Skipped 3 previous similar messages [ 670.562538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 681.258851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 681.288685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 681.316113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 684.557978] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:45:52 (1773668752) [ 684.790304] LNet: 69724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.792891] LNetError: 69724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 684.796053] LNetError: 69724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 684.799446] LNet: Removed LNI 192.168.204.59@tcp [ 684.805924] LNet: Removed LNI 192.168.204.59@tcp1 [ 685.093615] Key type .llcrypt unregistered [ 685.095798] Key type ._llcrypt unregistered [ 687.740651] Key type ._llcrypt registered [ 687.743200] Key type .llcrypt registered [ 687.874498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 687.885769] alg: No test for adler32 (adler32-zlib) [ 688.716969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 688.737523] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 693.790717] Lustre: DEBUG MARKER: Initial discovery [ 693.841743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.159@tcp [ 695.005521] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 695.051365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 695.084527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.159@tcp [ 695.105562] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 695.119560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 695.148635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.159@tcp -r 1 -m GET -l 3 [ 695.175939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.159@tcp -r 1 -m GET -e local_error [ 695.203532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.159@tcp -r 1 -m PUT -l 6 [ 695.906517] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 696.071503] Lustre: DEBUG MARKER: Force oleg459-client.virtnet to discover 192.168.204.159@tcp (in background) [ 696.121398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.159@tcp [ 696.937267] Lustre: DEBUG MARKER: Force oleg459-server to discover 192.168.204.59@tcp [ 698.175823] Lustre: DEBUG MARKER: Wait for 70479 [ 699.168852] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 699.918687] Lustre: DEBUG MARKER: Finished wait on 70479 [ 699.973705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 700.006118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 700.184852] LNet: 70935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.195459] LNetError: 70935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.227783] LNet: Removed LNI 192.168.204.59@tcp [ 701.511711] Key type .llcrypt unregistered [ 701.513787] Key type ._llcrypt unregistered [ 709.923487] Key type ._llcrypt registered [ 709.924992] Key type .llcrypt registered [ 710.636770] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:46:18 (1773668778) [ 710.935803] Key type .llcrypt unregistered [ 710.938633] Key type ._llcrypt unregistered [ 714.263280] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 714.347315] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 714.351648] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 714.355564] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 716.688616] Key type ._llcrypt registered [ 716.690537] Key type .llcrypt registered [ 716.819592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 716.828461] alg: No test for adler32 (adler32-zlib) [ 717.703150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.743945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 717.760692] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 718.246495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 718.261153] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 723.244092] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:46:31 (1773668791) [ 723.291068] LNet: 72998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.296904] LNetError: 72998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 723.302882] LNet: Removed LNI 192.168.204.59@tcp [ 723.319755] LNet: Removed LNI 10.1.2.3@tcp [ 723.789108] Key type .llcrypt unregistered [ 723.793127] Key type ._llcrypt unregistered [ 727.387081] Key type ._llcrypt registered [ 727.388371] Key type .llcrypt registered [ 727.499213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 727.508765] alg: No test for adler32 (adler32-zlib) [ 728.365718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.394785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 728.407090] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 728.843708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 728.860603] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 729.431462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.59@tcp [ 730.464581] LNetError: Unexpected error -22 connecting to 192.168.204.59@tcp at host 192.168.204.59:988 [ 733.134423] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:46:41 (1773668801) [ 733.209616] LNet: 74132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.212739] LNetError: 74132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 733.215709] LNet: Removed LNI 192.168.204.59@tcp [ 733.228321] LNet: Removed LNI 10.1.2.3@tcp [ 733.549901] Key type .llcrypt unregistered [ 733.551953] Key type ._llcrypt unregistered [ 736.975477] Key type ._llcrypt registered [ 736.977222] Key type .llcrypt registered [ 737.147972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 737.154699] alg: No test for adler32 (adler32-zlib) [ 737.993193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.021898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 738.039930] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 738.453737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 738.473438] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 739.075585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.59@tcp1 --nid 192.168.204.59@tcp2 [ 741.437477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp1 192.168.204.59@tcp2 [ 741.453455] LNetError: 74971:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.59@tcp1 to 192.168.204.59@tcp2 - Route table may be misconfigured [ 741.466906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp2 192.168.204.59@tcp1 [ 744.969602] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:46:53 (1773668813) [ 745.045840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.060493] LNet: 75541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.065298] LNetError: 75541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.080273] LNet: Removed LNI 192.168.204.59@tcp1 [ 746.083207] LNetError: 75541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.095431] LNet: Removed LNI 192.168.204.59@tcp2 [ 746.114711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.161592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 746.175653] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 746.529770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 746.544796] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 747.057115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 747.083529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp 192.168.204.59@tcp [ 747.103143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp1 192.168.204.59@tcp1 [ 747.129971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.59@tcp -d 192.168.204.59@tcp -e network_timeout [ 747.150023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -e network_timeout [ 747.173338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -e network_timeout [ 747.194171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -e network_timeout [ 747.215981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 747.230185] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 747.243133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 747.304823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 747.388912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 748.193538] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 751.091711] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:46:58 (1773668818) [ 751.174956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 751.192717] LNet: 76398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.200540] LNetError: 76398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 751.204556] LNet: Removed LNI 192.168.204.59@tcp [ 752.232923] LNet: Removed LNI 192.168.204.59@tcp1 [ 752.265923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.363456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 752.581232] LNet: 76442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.833657] Key type .llcrypt unregistered [ 752.836801] Key type ._llcrypt unregistered [ 759.292887] Key type ._llcrypt registered [ 759.294374] Key type .llcrypt registered [ 759.963329] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:47:08 (1773668828) [ 760.185700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 760.197295] alg: No test for adler32 (adler32-zlib) [ 761.020474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.792988] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 762.712572] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:47:10 (1773668830) [ 762.770769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.788525] LNet: 77533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.810403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.846644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 762.858193] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 763.205179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 763.219153] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 763.691829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 763.713317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp1 [ 763.736550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp1 [ 767.068616] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:47:15 (1773668835) [ 768.075509] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg459-server [ 769.190787] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:47:17 (1773668837) [ 770.459925] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg459-server [ 771.454150] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:47:19 (1773668839) [ 772.604369] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg459-server [ 773.629775] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:47:21 (1773668841) [ 774.776520] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg459-server [ 775.698771] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:47:23 (1773668843) [ 776.699267] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg459-server [ 777.743699] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:47:25 (1773668845) [ 778.748850] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg459-server [ 779.687847] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:47:27 (1773668847) [ 780.740437] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg459-server [ 781.749417] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:47:29 (1773668849) [ 782.624217] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg459-server [ 783.777847] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:47:31 (1773668851) [ 784.815600] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg459-server [ 785.680425] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:47:33 (1773668853) [ 785.731211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.744535] LNet: 81408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.748485] LNetError: 81408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.752840] LNet: Removed LNI 192.168.204.59@tcp [ 786.791287] LNet: Removed LNI 192.168.204.59@tcp1 [ 786.801187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.829695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 786.848234] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 787.181179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 787.220268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 787.324518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.334796] LNet: 81560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.338056] LNetError: 81560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.341518] LNetError: 81560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 787.352695] LNet: Removed LNI 192.168.204.59@tcp [ 787.363442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.390801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 787.404817] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 787.808754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 787.911972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 788.967303] LNet: Removed LNI 192.168.204.59@tcp [ 788.983138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.021782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 789.041127] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 789.478262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 789.528340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 789.628298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 789.637518] LNet: 81852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.640672] LNet: 81852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 789.643896] LNetError: 81852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.646593] LNetError: 81852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 789.660858] LNet: Removed LNI 192.168.204.59@tcp [ 789.671871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.697935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 789.711944] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 790.097326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 790.209970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.271236] LNet: Removed LNI 192.168.204.59@tcp [ 791.285654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.328138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 791.344531] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 791.727645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 791.770850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 791.896186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.910979] LNetError: 82144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.915762] LNetError: 82144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 791.931262] LNet: Removed LNI 192.168.204.59@tcp [ 791.942848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.991634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 792.013665] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 792.426604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 792.554979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 793.639483] LNet: Removed LNI 192.168.204.59@tcp [ 793.652667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 793.700704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 793.716964] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 794.120291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 794.170712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 794.293035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.305958] LNet: 82437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.312345] LNet: 82437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 795.366879] LNet: Removed LNI 192.168.204.59@tcp [ 795.380308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.436790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 795.452268] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 795.901226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 796.031816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.048097] LNetError: 82582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 796.053730] LNetError: 82582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 796.065729] LNet: Removed LNI 192.168.204.59@tcp [ 796.078931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.122347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 796.148426] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 796.545383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 796.592690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 796.729635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.800870] LNet: Removed LNI 192.168.204.59@tcp [ 797.815933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.848243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 797.863887] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 798.305456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 798.443437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.528164] LNet: Removed LNI 192.168.204.59@tcp [ 799.547539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.580184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.596139] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 800.024167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 800.083148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 800.275347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.320367] LNet: Removed LNI 192.168.204.59@tcp [ 801.341969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.385076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 801.397292] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 801.875929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 802.050124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.118176] LNet: Removed LNI 192.168.204.59@tcp [ 803.141709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.180987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 803.201060] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 803.649972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 803.690266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.705113] LNet: 83299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.710169] LNet: 83299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 803.721756] LNet: Removed LNI 192.168.204.59@tcp [ 803.736716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.783431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 803.803069] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 804.217275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 808.367223] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:47:56 (1773668876) [ 808.445363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.460507] LNetError: 83996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.466115] LNetError: 83996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 808.476753] LNet: Removed LNI 192.168.204.59@tcp [ 808.490934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.526548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 808.540601] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/180] [ 809.213846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.243269] LNet: Removed LNI 192.168.204.59@tcp231 [ 809.251404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.301780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 809.323291] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/180] [ 809.332417] LNet: 84149:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 809.689495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 809.710222] LNet: Removed LNI 192.168.204.59@tcp231 [ 809.723566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 809.746586] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/180] [ 810.395261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.428855] LNet: Removed LNI 192.168.204.59@tcp231 [ 810.442359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.489967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 810.507856] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/0] [ 810.512472] LNet: 84301:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 810.760562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 810.787663] LNet: Removed LNI 192.168.204.59@tcp231 [ 810.799600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 810.812055] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/0] [ 811.438607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.464502] LNet: Removed LNI 192.168.204.59@tcp231 [ 811.473755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.500424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 811.515294] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/60] [ 811.519368] LNet: 84453:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 811.749567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 811.772553] LNet: Removed LNI 192.168.204.59@tcp231 [ 811.779712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 811.801204] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/60] [ 812.408691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.439813] LNet: Removed LNI 192.168.204.59@tcp231 [ 812.451886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.500677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 812.514798] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/180] [ 812.713578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 812.739788] LNet: Removed LNI 192.168.204.59@tcp231 [ 812.751732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 812.774452] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/180] [ 816.756389] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:48:04 (1773668884) [ 816.879497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.901590] LNet: Removed LNI 192.168.204.59@tcp231 [ 816.910175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.936982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 816.951338] LNet: Added LNI 192.168.204.59@tcp232 [8/256/0/180] [ 817.350166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 817.369179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 817.400153] LNet: Removed LNI 192.168.204.59@tcp232 [ 817.410948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 817.428887] LNet: Added LNI 192.168.204.59@tcp232 [8/256/0/180] [ 817.931375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 817.959244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 817.988475] LNet: Removed LNI 192.168.204.59@tcp232 [ 817.997622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 818.013047] LNet: Added LNI 192.168.204.59@tcp232 [8/256/0/180] [ 818.431703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 818.461465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 818.495465] LNet: Removed LNI 192.168.204.59@tcp232 [ 818.508723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 818.529618] LNet: Added LNI 192.168.204.59@tcp232 [8/256/0/180] [ 818.957955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 818.990982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 819.021986] LNet: Removed LNI 192.168.204.59@tcp232 [ 819.037898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 823.131355] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:48:11 (1773668891) [ 823.435539] LNet: 86441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.439160] LNet: 86441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 823.887201] Key type .llcrypt unregistered [ 823.890125] Key type ._llcrypt unregistered [ 827.194350] Key type ._llcrypt registered [ 827.196213] Key type .llcrypt registered [ 827.331942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 827.345281] alg: No test for adler32 (adler32-zlib) [ 828.233032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 828.252847] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 832.387283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 836.692096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.716920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 836.735443] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 837.542065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_interrupt -r 2 [ 837.563564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_interrupt -r 2 [ 837.585479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_interrupt -r 2 [ 837.610773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_interrupt -r 2 [ 837.632055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_interrupt -r 2 [ 837.651741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_interrupt -r 2 [ 837.676773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_interrupt -r 2 [ 837.699539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_interrupt -r 2 [ 837.726396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp 192.168.204.159@tcp [ 837.745046] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 837.754915] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 837.907641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 837.928536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 837.947879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 837.974276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 837.999411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 838.028184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 838.053598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 838.076148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 838.086876] LNet: 87101:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 838.102212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 838.125464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 838.149962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 838.172192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 838.197709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 838.229243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 838.519404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_dropped -r 2 [ 838.548535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_dropped -r 2 [ 838.577347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_dropped -r 2 [ 838.605790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_dropped -r 2 [ 838.624819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_dropped -r 2 [ 838.648629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_dropped -r 2 [ 838.668313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_dropped -r 2 [ 838.693728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_dropped -r 2 [ 838.729429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp 192.168.204.159@tcp [ 838.743641] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 838.887595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 838.908825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 838.928341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 838.945960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 838.970740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 838.984726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 839.001603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 839.023478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 839.039478] LNet: 87221:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 839.054339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 839.074902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 839.095498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 839.111521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 839.133268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 839.155254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 839.460301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_aborted -r 2 [ 839.480382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_aborted -r 2 [ 839.504171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_aborted -r 2 [ 839.534861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_aborted -r 2 [ 839.558399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_aborted -r 2 [ 839.581873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_aborted -r 2 [ 839.605699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_aborted -r 2 [ 839.627630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_aborted -r 2 [ 839.644548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp 192.168.204.159@tcp [ 839.772979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 839.789582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 839.810713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 839.828151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 839.843279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 839.857678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 839.878338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 839.893899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 839.909631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 839.929967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 839.959856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 839.980816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 840.004599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 840.033360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 840.361852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_no_route -r 2 [ 840.387906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_no_route -r 2 [ 840.405123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_no_route -r 2 [ 840.422447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_no_route -r 2 [ 840.439623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_no_route -r 2 [ 840.461415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_no_route -r 2 [ 840.481640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_no_route -r 2 [ 840.504765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_no_route -r 2 [ 840.537653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp 192.168.204.159@tcp [ 840.557960] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 840.561239] LNet: Skipped 2 previous similar messages [ 840.708566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 840.732155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 840.757710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 840.786566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 840.805784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 840.823474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 840.845565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 840.859421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 840.872346] LNet: 87461:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 840.876582] LNet: 87461:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 840.885654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 840.903511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 840.924674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 840.948420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 840.965363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 840.983931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 841.255631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_timeout -r 2 [ 841.292313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_timeout -r 2 [ 841.317996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_timeout -r 2 [ 841.346663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_timeout -r 2 [ 841.372703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_timeout -r 2 [ 841.400283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_timeout -r 2 [ 841.428592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_timeout -r 2 [ 841.466325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_timeout -r 2 [ 841.494066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp 192.168.204.159@tcp [ 841.655798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 841.674801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 841.696895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 841.718993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 841.742388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 841.761104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 841.776220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 841.792345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 841.811320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 841.830272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 841.848586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 841.869705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 841.890126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 841.922778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 842.209889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e remote_dropped -r 2 [ 842.237052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e remote_dropped -r 2 [ 842.256807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e remote_dropped -r 2 [ 842.277342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e remote_dropped -r 2 [ 842.298485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e remote_dropped -r 2 [ 842.319126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e remote_dropped -r 2 [ 842.340766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e remote_dropped -r 2 [ 842.360872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e remote_dropped -r 2 [ 842.380469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp 192.168.204.159@tcp [ 842.397107] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.159@tcp [ 842.528369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 842.542214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 842.557389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 842.577338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 842.603297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 842.635347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 842.661361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 842.683803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 842.696951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 842.713819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 842.729688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 842.752226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 842.782288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 842.807762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 845.713663] LNet: 87775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.718045] LNetError: 87775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.723285] LNet: Removed LNI 192.168.204.59@tcp [ 845.730181] LNet: Removed LNI 192.168.204.59@tcp1 [ 845.939170] Key type .llcrypt unregistered [ 845.940839] Key type ._llcrypt unregistered [ 854.217298] Key type ._llcrypt registered [ 854.219971] Key type .llcrypt registered [ 854.904857] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 09:48:42 (1773668922) [ 855.078345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 855.087145] alg: No test for adler32 (adler32-zlib) [ 855.930220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.963141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 855.984903] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 856.467373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 856.486616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.59@tcp [ 856.501695] LNetError: 88752:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.59@tcp to recovery [ 856.551283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.59@tcp [ 856.602271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 859.823432] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 09:48:47 (1773668927) [ 859.899520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.911932] LNet: 89338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.915467] LNetError: 89338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.929452] LNet: Removed LNI 192.168.204.59@tcp [ 859.938988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.961330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 859.972182] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 860.350601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 860.362878] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 860.890409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.59@tcp [ 860.899701] LNetError: 89607:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.59@tcp to recovery [ 860.960376] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 860.962995] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 862.023853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.59@tcp [ 865.996578] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 09:48:54 (1773668934) [ 867.154476] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg459-server [ 867.905450] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 09:48:56 (1773668936) [ 868.774792] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg459-server [ 869.653798] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 09:48:57 (1773668937) [ 870.582293] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg459-server [ 871.478641] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:48:59 (1773668939) [ 871.647857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.661342] LNet: 91228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.665879] LNetError: 91228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.672134] LNet: Removed LNI 192.168.204.59@tcp [ 872.674679] LNetError: 91228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.687371] LNet: Removed LNI 192.168.204.59@tcp1 [ 872.703768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.745298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 872.761455] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 873.163633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 873.196344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 873.231666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 873.288688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 873.320321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 873.356979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 876.678282] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 09:49:04 (1773668944) [ 876.808859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.822312] LNet: 91953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.826270] LNetError: 91953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.840216] LNet: Removed LNI 192.168.204.59@tcp [ 876.860757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.907415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 876.925332] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 881.036198] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:49:09 (1773668949) [ 881.093166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.108107] LNet: 92648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.113830] LNetError: 92648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.123218] LNet: Removed LNI 192.168.204.59@tcp [ 881.136238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.173574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 881.186096] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 885.109922] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:49:13 (1773668953) [ 886.003619] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 887.162335] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:49:14 (1773668954) [ 887.431061] LNet: 93689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.441206] LNet: Removed LNI 192.168.204.59@tcp [ 887.813437] Key type .llcrypt unregistered [ 887.816042] Key type ._llcrypt unregistered [ 890.887305] Key type ._llcrypt registered [ 890.889652] Key type .llcrypt registered [ 891.012773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 891.021630] alg: No test for adler32 (adler32-zlib) [ 891.872897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 891.893979] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 895.915569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 898.008333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.159@tcp [ 898.207880] LNet: 94106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.212192] LNetError: 94106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.227747] LNet: Removed LNI 192.168.204.59@tcp [ 898.413597] Key type .llcrypt unregistered [ 898.415260] Key type ._llcrypt unregistered [ 903.231379] Key type ._llcrypt registered [ 903.232738] Key type .llcrypt registered [ 903.924805] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 904.746469] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 905.741594] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:49:33 (1773668973) [ 906.007899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 906.017476] alg: No test for adler32 (adler32-zlib) [ 906.879557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.938470] LNet: 95193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.335349] Key type .llcrypt unregistered [ 907.336882] Key type ._llcrypt unregistered [ 910.196441] Key type ._llcrypt registered [ 910.198747] Key type .llcrypt registered [ 910.306502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 910.326663] alg: No test for adler32 (adler32-zlib) [ 911.151136] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 911.153106] LNet: 95449:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 911.157274] LNet: 95449:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 911.431785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.601342] LNetError: 95461:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.204.63@tcp has gone from up to down [ 917.611973] LNet: 95507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.617704] LNetError: 95507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.634178] LNet: Removed LNI 192.168.204.59@tcp [ 917.652578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.689713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 917.706077] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 917.711346] LNet: 95515:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 917.719933] LNet: 95515:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 917.725303] LNetError: 95515:0:(peer.c:366:lnet_peer_ni_del_locked()) Peer NI 192.168.204.60@tcp is a gateway. Can not delete it [ 917.734897] LNetError: 95515:0:(peer.c:2166:lnet_del_peer_ni()) 192.168.204.60@tcp is a router. Can not be deleted [ 917.751440] LNet: 95515:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 920.422336] LNetError: 95516:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.204.64@tcp has gone from up to down [ 920.433733] LNetError: 95516:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 921.291698] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:49:49 (1773668989) [ 922.224320] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg459-server [ 923.140308] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 09:49:51 (1773668991) [ 924.078476] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg459-server [ 924.825203] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:49:53 (1773668993) [ 925.517708] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 927.248170] Key type lgssc registered [ 927.748983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 936.792669] Key type lgssc unregistered [ 936.977456] LNet: 97933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.980364] LNetError: 97933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.992717] LNet: Removed LNI 192.168.204.59@tcp [ 937.270750] Key type .llcrypt unregistered [ 937.272108] Key type ._llcrypt unregistered [ 943.431101] Key type ._llcrypt registered [ 943.432342] Key type .llcrypt registered [ 944.035396] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 09:50:12 (1773669012) [ 944.190504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 944.196791] alg: No test for adler32 (adler32-zlib) [ 945.008894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.040562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 945.064356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 945.084678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 945.106387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 947.825720] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 09:50:16 (1773669016) [ 947.893311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.904354] LNet: 99334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.918648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.959235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 947.971892] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 948.380578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.392996] LNetError: 99481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.403519] LNet: Removed LNI 192.168.204.59@tcp [ 948.412309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.441277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 948.454323] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 948.654551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.665345] LNet: 99509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.667954] LNet: 99509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 948.678364] LNet: Removed LNI 192.168.204.59@tcp [ 948.692213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.717753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 948.732720] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 952.057837] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 09:50:20 (1773669020) [ 952.111248] LNet: 100095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.117668] LNetError: 100095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.124818] LNetError: 100095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 952.137406] LNet: Removed LNI 192.168.204.59@tcp [ 952.451668] Key type .llcrypt unregistered [ 952.453417] Key type ._llcrypt unregistered [ 955.444715] Key type ._llcrypt registered [ 955.449593] Key type .llcrypt registered [ 955.605675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 955.624967] alg: No test for adler32 (adler32-zlib) [ 956.455288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.496280] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 956.498494] LNet: *** cfs_fail_loc=e003, val=100*** [ 956.936278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 956.949952] Ping reply received with mapping: [ 956.951733] found metadata reply NI[192.168.204.59@tcp] [ 956.954863] 3b cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 956.957035] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 956.959250] 5a 5a 5a 5a 5a 5a 5a 5a [ 957.016153] LNet: 100497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.019190] LNetError: 100497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.030561] LNet: Removed LNI 192.168.204.59@tcp [ 957.357464] Key type .llcrypt unregistered [ 957.358496] Key type ._llcrypt unregistered [ 962.336750] Key type ._llcrypt registered [ 962.338613] Key type .llcrypt registered [ 963.103818] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:50:31 (1773669031) [ 963.332785] Key type .llcrypt unregistered [ 963.335686] Key type ._llcrypt unregistered [ 965.900440] Key type ._llcrypt registered [ 965.902598] Key type .llcrypt registered [ 966.013208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 966.022472] alg: No test for adler32 (adler32-zlib) [ 966.948829] LNetError: 101556:0:(api-ni.c:2731:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 967.119690] Key type .llcrypt unregistered [ 967.121984] Key type ._llcrypt unregistered [ 969.381071] Key type ._llcrypt registered [ 969.382731] Key type .llcrypt registered [ 969.940922] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 09:50:38 (1773669038) [ 970.155441] Key type .llcrypt unregistered [ 970.157062] Key type ._llcrypt unregistered [ 972.513158] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 972.519344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 974.746461] Key type ._llcrypt registered [ 974.747988] Key type .llcrypt registered [ 974.833499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 974.841728] alg: No test for adler32 (adler32-zlib) [ 975.677506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.706066] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 980.476500] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 09:50:48 (1773669048) [ 980.528469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.551610] LNet: 103328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.556592] LNetError: 103328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 980.565569] LNet: Removed LNI 10.1.2.3@tcp [ 980.584991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.956934] Key type .llcrypt unregistered [ 980.961090] Key type ._llcrypt unregistered [ 983.505420] Key type ._llcrypt registered [ 983.506474] Key type .llcrypt registered [ 983.591655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 983.597418] alg: No test for adler32 (adler32-zlib) [ 984.422230] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 984.454802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 984.523160] LNet: 103622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.526551] LNetError: 103622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.537898] LNet: Removed LNI 192.168.204.59@tcp [ 984.826326] Key type .llcrypt unregistered [ 984.827569] Key type ._llcrypt unregistered [ 989.738889] Key type ._llcrypt registered [ 989.740206] Key type .llcrypt registered [ 990.350882] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:50:58 (1773669058) [ 990.736334] Key type .llcrypt unregistered [ 990.737675] Key type ._llcrypt unregistered [ 993.887398] Key type ._llcrypt registered [ 993.892613] Key type .llcrypt registered [ 994.036574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 994.046890] alg: No test for adler32 (adler32-zlib) [ 997.914941] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:51:06 (1773669066) [ 997.961843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.995887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.024051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 998.043606] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 998.430181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 998.452241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 998.473163] LNet: Removed LNI 192.168.204.59@tcp [ 998.569278] LNet: 105364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.748561] Key type .llcrypt unregistered [ 998.750118] Key type ._llcrypt unregistered [ 1003.389923] Key type ._llcrypt registered [ 1003.391800] Key type .llcrypt registered [ 1003.966401] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:51:12 (1773669072) [ 1004.124881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1004.131520] alg: No test for adler32 (adler32-zlib) [ 1004.944133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.978735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1004.990431] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1005.353391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1005.377813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.59@tcp [ 1005.387374] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1005.389421] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1005.391081] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1005, 0, 0 [ 1005.395145] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1005.396906] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1005.398969] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1005.400837] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1005, 0, 0 [ 1005.403624] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1005.405227] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1005.410027] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1005.412835] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1005, 0, 1055 [ 1005.416757] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1005.418395] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1005.421014] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1005.422418] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1005, 0, 1055 [ 1005.424871] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1008.079612] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:51:16 (1773669076) [ 1008.286107] LNet: 106903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.290553] LNetError: 106903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.301571] LNet: Removed LNI 192.168.204.59@tcp [ 1008.604362] Key type .llcrypt unregistered [ 1008.605986] Key type ._llcrypt unregistered [ 1011.092602] Key type ._llcrypt registered [ 1011.094355] Key type .llcrypt registered [ 1011.179534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1011.186454] alg: No test for adler32 (adler32-zlib) [ 1012.016469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1012.031858] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1015.616662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 1019.243679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1019.270776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1019.284184] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 1019.706765] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1019.748417] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1019.750733] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1019.754066] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1021.958708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1021.969430] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1026.647883] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:51:34 (1773669094) [ 1026.695956] LNet: 108403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.700359] LNetError: 108403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.704079] LNet: Removed LNI 192.168.204.59@tcp [ 1027.744296] LNet: Removed LNI 192.168.204.59@tcp1 [ 1027.746044] LNetError: 108403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1027.749151] LNetError: 108403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1027.760485] LNet: Removed LNI 10.1.2.3@tcp99 [ 1028.044947] Key type .llcrypt unregistered [ 1028.049276] Key type ._llcrypt unregistered [ 1030.780955] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1030.784350] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1033.108719] Key type ._llcrypt registered [ 1033.111033] Key type .llcrypt registered [ 1033.199886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1033.206757] alg: No test for adler32 (adler32-zlib) [ 1034.027284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.060328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1034.074156] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1034.430641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1034.440723] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1035.504077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1035.526221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1035.545050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1035.555103] LNetError: 109153:0:(peer.c:2174:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1035.571017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1035.594833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1035.788052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1035.843903] LNet: 109173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.850946] LNetError: 109173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.854857] LNet: Removed LNI 192.168.204.59@tcp [ 1035.861677] LNet: Removed LNI 10.1.2.3@tcp [ 1036.214434] Key type .llcrypt unregistered [ 1036.216906] Key type ._llcrypt unregistered [ 1041.265290] Key type ._llcrypt registered [ 1041.266869] Key type .llcrypt registered [ 1042.032181] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:51:49 (1773669109) [ 1042.250423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1042.263708] alg: No test for adler32 (adler32-zlib) [ 1043.098390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.134149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1043.147577] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1043.563644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1043.606656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg459-client@tcp [ 1046.712986] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:51:54 (1773669114) [ 1046.744402] LNet: 110695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.748914] LNetError: 110695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.766443] LNet: Removed LNI 192.168.204.59@tcp [ 1047.084761] Key type .llcrypt unregistered [ 1047.087370] Key type ._llcrypt unregistered [ 1049.734525] Key type ._llcrypt registered [ 1049.738482] Key type .llcrypt registered [ 1049.829886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1049.837883] alg: No test for adler32 (adler32-zlib) [ 1050.657166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1050.667249] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1050.685659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1053.602252] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:52:01 (1773669121) [ 1054.274281] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1055.233314] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:52:03 (1773669123) [ 1056.007171] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1056.839917] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 09:52:04 (1773669124) [ 1056.902742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1056.914372] LNet: 112115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.918283] LNetError: 112115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1056.928131] LNet: Removed LNI 192.168.204.59@tcp [ 1056.943883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1056.992057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1057.010686] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1057.363223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1057.391288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1060.702992] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:52:08 (1773669128) [ 1060.839574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1060.854144] LNet: 112831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.856895] LNetError: 112831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.930333] LNet: Removed LNI 192.168.204.59@tcp [ 1061.952222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.992448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1062.014045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1062.162261] LNet: 112867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.376371] Key type .llcrypt unregistered [ 1062.381908] Key type ._llcrypt unregistered [ 1067.212669] Key type ._llcrypt registered [ 1067.213897] Key type .llcrypt registered [ 1067.801548] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:52:15 (1773669135) [ 1068.013539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1068.019387] alg: No test for adler32 (adler32-zlib) [ 1068.842413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1068.874280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1068.887444] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1069.208201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1069.235314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1072.369576] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:52:20 (1773669140) [ 1072.420893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1072.431223] LNet: 114383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.433939] LNetError: 114383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1072.441690] LNet: Removed LNI 192.168.204.59@tcp [ 1072.453739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1072.483119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1072.506353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1075.419292] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 09:52:23 (1773669143) [ 1075.455851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1075.467796] LNet: 114958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.482432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1075.521928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1075.691084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1075.709932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1075.737911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1075.920428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1075.942471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1075.993195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1076.019670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1076.061132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1076.259179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1076.281581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1076.322910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1076.507502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1076.541668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1079.897596] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 09:52:27 (1773669147) [ 1079.988438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1080.005097] LNet: 115596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.009760] LNet: 115596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1080.027691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1080.068751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1080.079570] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1080.521655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1080.532646] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 1081.050734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1081.078435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1081.215744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1081.227803] LNetError: 115876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1082.272137] LNet: Removed LNI 192.168.204.59@tcp [ 1082.273736] LNetError: 115876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1082.281481] LNet: Removed LNI 192.168.204.59@tcp2 [ 1082.290375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1082.317175] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1082.320627] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 1082.323717] LNet: 115882:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1082.516262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1082.597362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1082.607735] LNet: 115919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.610798] LNet: 115919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1083.616135] LNet: Removed LNI 192.168.204.59@tcp [ 1083.624337] LNet: Removed LNI 192.168.204.59@tcp2 [ 1083.641106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.673243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1083.687880] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1084.057186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1084.066975] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 1084.457292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1084.477342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1084.506290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1084.635382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1084.645774] LNetError: 116203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.648899] LNetError: 116203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1084.652700] LNet: Removed LNI 192.168.204.59@tcp [ 1085.670579] LNet: Removed LNI 192.168.204.59@tcp2 [ 1085.680164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.717554] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1085.722402] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 1085.725234] LNet: 116209:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1085.926224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1086.028318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1086.041482] LNet: Removed LNI 192.168.204.59@tcp [ 1087.080509] LNet: Removed LNI 192.168.204.59@tcp2 [ 1087.103465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1087.134211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1087.145261] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1087.453613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1087.463269] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 1087.845653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1087.872432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1087.962534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1087.972399] LNet: 116526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.975709] LNet: 116526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1088.992555] LNet: Removed LNI 192.168.204.59@tcp [ 1088.994297] LNetError: 116526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1088.997121] LNetError: 116526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1089.005306] LNet: Removed LNI 192.168.204.59@tcp2 [ 1089.015640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1089.045690] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1089.048608] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 1089.050866] LNet: 116532:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1089.246809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1089.324625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1090.400235] LNet: Removed LNI 192.168.204.59@tcp [ 1090.409121] LNet: Removed LNI 192.168.204.59@tcp2 [ 1090.429703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1090.457319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1090.469670] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1090.804195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1090.815633] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 1091.252525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1091.274268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1091.298744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1091.390988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1092.449476] LNet: Removed LNI 192.168.204.59@tcp [ 1093.479728] LNet: Removed LNI 192.168.204.59@tcp2 [ 1093.491512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1093.528960] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1093.533234] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 1093.540448] LNet: 116860:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1093.747571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1093.834721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1093.845995] LNet: Removed LNI 192.168.204.59@tcp [ 1094.889196] LNet: Removed LNI 192.168.204.59@tcp2 [ 1094.902797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1094.936859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1094.949075] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1095.291923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1095.302848] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 1095.713262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1095.731105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1095.757294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1095.904780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1096.928520] LNet: Removed LNI 192.168.204.59@tcp [ 1096.936443] LNet: Removed LNI 192.168.204.59@tcp2 [ 1096.947715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1096.978695] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1096.982297] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 1097.191527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1097.340868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1097.352408] LNet: 117251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.357312] LNet: 117251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1097.363510] LNetError: 117251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.368414] LNetError: 117251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1097.373648] LNet: Removed LNI 192.168.204.59@tcp [ 1097.387068] LNet: Removed LNI 192.168.204.59@tcp2 [ 1097.397088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1097.429420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1097.444144] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1097.790834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1097.802386] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 1098.202110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1098.221533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1098.240316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1098.268533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1098.428648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1099.488360] LNet: Removed LNI 192.168.204.59@tcp [ 1100.518161] LNet: Removed LNI 192.168.204.59@tcp2 [ 1100.528618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1100.560600] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1100.563569] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 1100.565813] LNet: 117559:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.569342] LNet: 117559:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1100.774302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1103.650567] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1104.385521] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 09:52:52 (1773669172) [ 1104.451372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1104.464737] LNet: Removed LNI 192.168.204.59@tcp [ 1105.510890] LNet: Removed LNI 192.168.204.59@tcp2 [ 1105.520974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1105.659023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1105.671710] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1106.168317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1106.185094] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1106.776952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1106.798295] LNet: Removed LNI 192.168.204.59@tcp [ 1106.809520] LNet: Removed LNI 10.1.2.3@tcp [ 1106.831613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1106.888836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1106.905542] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1106.908747] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1107.130625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1107.147280] LNet: Removed LNI 192.168.204.59@tcp [ 1107.157805] LNet: Removed LNI 10.1.2.3@tcp [ 1111.805535] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:52:59 (1773669179) [ 1112.055638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1112.135074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1112.325405] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1112.457385] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1112.459919] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1112.461614] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1114.722123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1114.737082] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1115.587802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1115.608584] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1117.493223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1117.527292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1117.573867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.59@tcp -m PUT -e local_timeout -r 1 [ 1117.630290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1117.684724] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1117.687157] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1118.106388] LNetError: 119252:0:(lib-move.c:4969:lnet_parse()) 192.168.204.59@tcp, src 192.168.204.59@tcp: Dropping PUT (error -108 looking up sender) [ 1119.138278] LNet: 119711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.151733] LNet: 119711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1119.162932] LNetError: 119711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1119.281357] LNetError: 119711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1119.289422] LNet: Removed LNI 192.168.204.59@tcp [ 1119.308502] LNet: Removed LNI 10.1.2.3@tcp [ 1119.772926] Key type .llcrypt unregistered [ 1119.779497] Key type ._llcrypt unregistered [ 1134.364551] Key type ._llcrypt registered [ 1134.365896] Key type .llcrypt registered [ 1135.723027] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:53:23 (1773669203) [ 1136.108321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1136.122806] alg: No test for adler32 (adler32-zlib) [ 1138.715533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1139.664418] LNet: 120566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.217506] Key type .llcrypt unregistered [ 1140.219583] Key type ._llcrypt unregistered [ 1146.193411] Key type ._llcrypt registered [ 1146.194912] Key type .llcrypt registered [ 1146.465032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1146.505449] alg: No test for adler32 (adler32-zlib) [ 1147.534711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1147.579336] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1157.130511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 1157.652764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1158.612468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.159@tcp [ 1158.882979] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.159@tcp [ 1158.989848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1159.041753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1159.117748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1159.197539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1159.291291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1159.366681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1159.421606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1159.481752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1159.512536] LNet: 121017:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1159.549270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1159.621960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1159.711783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1159.776648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1159.853024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.920489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1160.917827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1161.291741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.351751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1161.396461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.450223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.516986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.576674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.644089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.707958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.745847] LNet: 121134:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1161.781047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1161.850345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1161.904464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.977245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.042842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.077167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.174211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.159@tcp [ 1163.563068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.621572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1163.663475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.737955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.781156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.822898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.870557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.906495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.922645] LNet: 121251:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.954589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.984864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1164.024254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1164.061856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1164.100880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1164.146044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.177939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1165.532510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1165.597066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1165.672198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.747829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.795910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.847666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.884816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.954983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.991499] LNet: 121369:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1166.018608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.067773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.110711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.157793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1166.232606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.333473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.032747] LNet: 121415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1167.058729] LNetError: 121415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1167.081242] LNet: Removed LNI 192.168.204.59@tcp [ 1167.775829] Key type .llcrypt unregistered [ 1167.779877] Key type ._llcrypt unregistered [ 1194.046398] Key type ._llcrypt registered [ 1194.048503] Key type .llcrypt registered [ 1196.095100] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 09:54:22 (1773669262) [ 1196.693863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1196.711880] alg: No test for adler32 (adler32-zlib) [ 1197.724891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1198.203420] LNet: 122305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.879722] Key type .llcrypt unregistered [ 1198.882962] Key type ._llcrypt unregistered [ 1205.678663] Key type ._llcrypt registered [ 1205.682885] Key type .llcrypt registered [ 1205.880723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1205.901719] alg: No test for adler32 (adler32-zlib) [ 1206.810636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1206.857819] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1214.669021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 1222.832725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1222.884479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1222.935195] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 1223.679249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1224.231775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.159@tcp [ 1224.296282] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.159@tcp [ 1224.409542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1224.436887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1224.464994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1224.491480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1224.526659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1224.579708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1224.611665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1224.643953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1224.662190] LNet: 122958:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1224.680422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1224.723273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1224.748820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1224.788165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1224.841802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1224.878780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1225.464684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1225.647608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1225.666703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1225.697759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1225.714032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1225.739462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1225.765569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1225.795788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1225.817876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1225.845683] LNet: 123075:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1225.880516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1225.916873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1225.938846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1225.966590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1225.990480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.021701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.658688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.159@tcp [ 1226.849737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1226.871076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1226.907768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.935240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.962897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1226.991736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1227.025960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1227.074220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1227.099340] LNet: 123193:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1227.124648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1227.165700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1227.203526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1227.235715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1227.272561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.319277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.923313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1228.180076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1228.208310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1228.237514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.275959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.314537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.345626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.399805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.434974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.465105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1228.504987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1228.545944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.578266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.615034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.662532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1233.455708] LNet: 123391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1233.463293] LNetError: 123391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1233.472990] LNet: Removed LNI 192.168.204.59@tcp [ 1233.486663] LNet: Removed LNI 192.168.204.59@tcp1 [ 1233.842774] Key type .llcrypt unregistered [ 1233.846031] Key type ._llcrypt unregistered [ 1248.925635] Key type ._llcrypt registered [ 1248.928371] Key type .llcrypt registered [ 1250.212414] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 09:55:17 (1773669317) [ 1256.795251] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 09:55:24 (1773669324) [ 1263.151658] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 09:55:30 (1773669330) [ 1270.380703] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 09:55:37 (1773669337) [ 1277.551750] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 09:55:45 (1773669345) [ 1283.806295] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 09:55:51 (1773669351) [ 1291.111432] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 09:55:58 (1773669358) [ 1299.197208] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 09:56:06 (1773669366) [ 1306.606442] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 09:56:14 (1773669374) [ 1313.144906] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 09:56:20 (1773669380) [ 1321.073683] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 09:56:28 (1773669388) [ 1328.044960] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 09:56:35 (1773669395) [ 1334.127536] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 09:56:41 (1773669401) [ 1339.506044] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 09:56:47 (1773669407) [ 1344.669668] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 09:56:52 (1773669412) [ 1349.813441] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 09:56:57 (1773669417) [ 1355.584508] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 09:57:03 (1773669423) [ 1361.560519] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 09:57:09 (1773669429) [ 1361.970464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1361.989702] alg: No test for adler32 (adler32-zlib) [ 1362.877648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.957706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1362.974415] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1363.777532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1363.793355] LNet: 134103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.798200] LNetError: 134103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.819063] LNet: Removed LNI 192.168.204.59@tcp [ 1363.838791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1363.893215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1363.916974] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1368.829779] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 09:57:16 (1773669436) [ 1373.896366] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 09:57:21 (1773669441) [ 1379.090281] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 09:57:26 (1773669446) [ 1384.310052] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 09:57:31 (1773669451) [ 1389.389251] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 09:57:36 (1773669456) [ 1394.632483] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 09:57:42 (1773669462) [ 1400.542300] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 09:57:48 (1773669468) [ 1406.547610] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 09:57:53 (1773669473) [ 1414.017300] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 09:58:01 (1773669481) [ 1421.286288] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 09:58:08 (1773669488) [ 1427.707981] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 09:58:15 (1773669495) [ 1433.279695] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 09:58:21 (1773669501) [ 1439.970642] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1312 sec ========== 09:58:27 (1773669507) [ 1440.297570] LNet: 141565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.301370] LNetError: 141565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1440.311422] LNet: Removed LNI 192.168.204.59@tcp [ 1440.923839] Key type .llcrypt unregistered [ 1440.926419] Key type ._llcrypt unregistered [ 1452.262576] Key type ._llcrypt registered [ 1452.270445] Key type .llcrypt registered [ 1452.838767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1452.855571] alg: No test for adler32 (adler32-zlib) [ 1453.916588] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 1454.166692] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1455.840265] Key type lgssc registered [ 1457.124752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1513.102676] Lustre: Mounted lustre-client [ 1517.668992] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1528.577772] LustreError: 144021:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e9e86a88800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1528.624085] Lustre: Unmounted lustre-client [ 1547.474120] Key type lgssc unregistered [ 1547.735113] LNet: 144502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1547.745141] LNetError: 144502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1547.758136] LNet: Removed LNI 192.168.204.59@tcp [ 1548.235198] Key type .llcrypt unregistered [ 1548.238198] Key type ._llcrypt unregistered