[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 598177843 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002451] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.006579] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010155] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013063] *** VALIDATE selinux *** [ 0.022308] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027192] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030093] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032061] *** VALIDATE tmpfs *** [ 0.034019] *** VALIDATE proc *** [ 0.035244] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.037261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039046] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041041] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.044793] debug: unmapping init [mem 0xffffffff96859000-0xffffffff96860fff] [ 0.046269] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047718] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048020] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055273] rcu: Hierarchical SRCU implementation. [ 0.057433] smp: Bringing up secondary CPUs ... [ 0.058653] x86: Booting SMP configuration: [ 0.059023] .... node #0, CPUs: #1 #2 #3 [ 0.066144] smp: Brought up 1 node, 4 CPUs [ 0.068014] smpboot: Max logical packages: 1 [ 0.069018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.103605] node 0 deferred pages initialised in 32ms [ 0.108035] devtmpfs: initialized [ 0.109393] x86/mm: Memory block size: 128MB [ 0.112146] gcov: version magic: 0x41383552 [ 0.114410] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.115084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.116240] pinctrl core: initialized pinctrl subsystem [ 0.117194] [ 0.117996] ************************************************************* [ 0.118029] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.119014] ** ** [ 0.120016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.121014] ** ** [ 0.122013] ** This means that this kernel is built to expose internal ** [ 0.123016] ** IOMMU data structures, which may compromise security on ** [ 0.124013] ** your system. ** [ 0.125016] ** ** [ 0.126013] ** If you see this message and you are not debugging the ** [ 0.127015] ** kernel, report this immediately to your vendor! ** [ 0.128014] ** ** [ 0.129014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.130013] ************************************************************* [ 0.131907] NET: Registered protocol family 16 [ 0.132533] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.133059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.134061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.135720] cpuidle: using governor menu [ 0.138031] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.141703] PCI: Using configuration type 1 for base access [ 0.143171] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.153116] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.154016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.157198] cryptd: max_cpu_qlen set to 1000 [ 0.160274] ACPI: Added _OSI(Module Device) [ 0.162019] ACPI: Added _OSI(Processor Device) [ 0.164014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.165012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.170780] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.178601] ACPI: Interpreter enabled [ 0.179082] ACPI: PM: (supports S0 S3 S4 S5) [ 0.180022] ACPI: Using IOAPIC for interrupt routing [ 0.181113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.182757] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.194036] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.195044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.196021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.197090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.199635] acpiphp: Slot [2] registered [ 0.200142] acpiphp: Slot [5] registered [ 0.201143] acpiphp: Slot [6] registered [ 0.202280] acpiphp: Slot [3] registered [ 0.203121] acpiphp: Slot [4] registered [ 0.204132] acpiphp: Slot [7] registered [ 0.205107] acpiphp: Slot [8] registered [ 0.206105] acpiphp: Slot [9] registered [ 0.207152] acpiphp: Slot [10] registered [ 0.208124] acpiphp: Slot [11] registered [ 0.209103] acpiphp: Slot [12] registered [ 0.210125] acpiphp: Slot [13] registered [ 0.211122] acpiphp: Slot [14] registered [ 0.212118] acpiphp: Slot [15] registered [ 0.213122] acpiphp: Slot [16] registered [ 0.214098] acpiphp: Slot [17] registered [ 0.215134] acpiphp: Slot [18] registered [ 0.216214] acpiphp: Slot [19] registered [ 0.217125] acpiphp: Slot [20] registered [ 0.218116] acpiphp: Slot [21] registered [ 0.219191] acpiphp: Slot [22] registered [ 0.220097] acpiphp: Slot [23] registered [ 0.221234] acpiphp: Slot [24] registered [ 0.222119] acpiphp: Slot [25] registered [ 0.223119] acpiphp: Slot [26] registered [ 0.224132] acpiphp: Slot [27] registered [ 0.225139] acpiphp: Slot [28] registered [ 0.226203] acpiphp: Slot [29] registered [ 0.227123] acpiphp: Slot [30] registered [ 0.228143] acpiphp: Slot [31] registered [ 0.229072] PCI host bridge to bus 0000:00 [ 0.230023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.231026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.232023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.233026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.234026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.235029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.236179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.238068] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.240299] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.248016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.253129] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.256020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.259016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.261018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.263723] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.266905] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.269170] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.272981] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.276632] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.285016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.289015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.324062] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.332023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.343017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414033] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.424553] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.432023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.447141] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.482022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.498056] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.501547] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.503510] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.506480] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.509245] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.515111] iommu: Default domain type: Passthrough [ 0.517555] SCSI subsystem initialized [ 0.519157] ACPI: bus type USB registered [ 0.521257] usbcore: registered new interface driver usbfs [ 0.523091] usbcore: registered new interface driver hub [ 0.525092] usbcore: registered new device driver usb [ 0.527181] pps_core: LinuxPPS API ver. 1 registered [ 0.529024] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.533079] PTP clock support registered [ 0.535418] EDAC MC: Ver: 3.0.0 [ 0.536775] PCI: Using ACPI for IRQ routing [ 0.537697] NetLabel: Initializing [ 0.539012] NetLabel: domain hash size = 128 [ 0.541011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.543097] NetLabel: unlabeled traffic allowed by default [ 0.587090] vgaarb: loaded [ 0.590457] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.596044] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.608354] clocksource: Switched to clocksource kvm-clock [ 0.831065] VFS: Disk quotas dquot_6.6.0 [ 0.833038] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.836345] *** VALIDATE ramfs *** [ 0.837935] *** VALIDATE hugetlbfs *** [ 0.840312] pnp: PnP ACPI init [ 0.846251] pnp: PnP ACPI: found 6 devices [ 0.868508] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.872864] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.876435] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.879910] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.883534] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.887611] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.891435] NET: Registered protocol family 2 [ 0.894262] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.899928] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.904132] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.910373] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.914814] TCP: Hash tables configured (established 65536 bind 65536) [ 0.918298] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.922204] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.925981] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.929885] NET: Registered protocol family 1 [ 0.932833] RPC: Registered named UNIX socket transport module. [ 0.935256] RPC: Registered udp transport module. [ 0.937918] RPC: Registered tcp transport module. [ 0.940213] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.943423] NET: Registered protocol family 44 [ 0.945478] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.948331] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.951113] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.953988] PCI: CLS 0 bytes, default 64 [ 0.955973] Unpacking initramfs... [ 3.002298] debug: unmapping init [mem 0xffff89383cc64000-0xffff89383ffcffff] [ 3.008451] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.010146] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.013460] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.617835] Initialise system trusted keyrings [ 3.619644] Key type blacklist registered [ 3.621456] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.632274] zbud: loaded [ 3.635217] *** VALIDATE nfs *** [ 3.636111] *** VALIDATE nfs4 *** [ 3.637547] pstore: using deflate compression [ 3.641035] Platform Keyring initialized [ 3.766517] NET: Registered protocol family 38 [ 3.768719] Key type asymmetric registered [ 3.770876] Asymmetric key parser 'x509' registered [ 3.773224] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.776929] io scheduler mq-deadline registered [ 3.778704] io scheduler kyber registered [ 3.780673] io scheduler bfq registered [ 3.782428] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.786055] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.789331] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.792978] ACPI: Power Button [PWRF] [ 3.799988] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.807899] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.831832] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.867642] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.908881] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.917150] Non-volatile memory driver v1.3 [ 3.918677] Linux agpgart interface v0.103 [ 3.954931] virtio_blk virtio1: [vda] 134040 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.958871] vda: detected capacity change from 0 to 68628480 [ 3.996172] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.000847] vdb: detected capacity change from 0 to 1073741824 [ 4.023861] libphy: Fixed MDIO Bus: probed [ 4.040111] usbcore: registered new interface driver usbserial_generic [ 4.043907] usbserial: USB Serial support registered for generic [ 4.047238] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.055979] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.059254] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.064136] mousedev: PS/2 mouse device common for all mice [ 4.070198] rtc_cmos 00:05: RTC can wake from S4 [ 4.074760] rtc_cmos 00:05: registered as rtc0 [ 4.077865] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.078956] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.079030] intel_pstate: CPU model not supported [ 4.084028] hid: raw HID events driver (C) Jiri Kosina [ 4.094515] usbcore: registered new interface driver usbhid [ 4.096356] usbhid: USB HID core driver [ 4.098707] drop_monitor: Initializing network drop monitor service [ 4.101784] Initializing XFRM netlink socket [ 4.105110] NET: Registered protocol family 10 [ 4.109718] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.112275] Segment Routing with IPv6 [ 4.117114] NET: Registered protocol family 17 [ 4.120897] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.121754] mpls_gso: MPLS GSO support [ 4.133258] RAS: Correctable Errors collector initialized. [ 4.135860] AVX version of gcm_enc/dec engaged. [ 4.137870] AES CTR mode by8 optimization enabled [ 4.230815] sched_clock: Marking stable (4230787318, 0)->(5356569217, -1125781899) [ 4.234197] registered taskstats version 1 [ 4.236136] Loading compiled-in X.509 certificates [ 4.237936] zswap: loaded using pool lzo/zbud [ 4.269121] Key type big_key registered [ 4.283277] Key type encrypted registered [ 4.284776] ima: No TPM chip found, activating TPM-bypass! [ 4.286776] ima: Allocated hash algorithm: sha1 [ 4.288332] ima: No architecture policies found [ 4.290175] evm: Initialising EVM extended attributes: [ 4.291750] evm: security.selinux [ 4.292892] evm: security.ima [ 4.293938] evm: security.capability [ 4.295111] evm: HMAC attrs: 0x1 [ 4.297718] rtc_cmos 00:05: setting system clock to 2026-01-22 09:15:10 UTC (1769073310) [ 4.305524] debug: unmapping init [mem 0xffffffff97803000-0xffffffff979fffff] [ 4.308430] debug: unmapping init [mem 0xffffffff96582000-0xffffffff96858fff] [ 4.318141] Write protecting the kernel read-only data: 28672k [ 4.321906] debug: unmapping init [mem 0xffffffff94c03000-0xffffffff94dfffff] [ 4.324571] debug: unmapping init [mem 0xffffffff95514000-0xffffffff955fffff] [ 4.359325] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.367425] systemd[1]: Detected virtualization kvm. [ 4.369212] systemd[1]: Detected architecture x86-64. [ 4.370843] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.401413] systemd[1]: No hostname configured. [ 4.403586] systemd[1]: Set hostname to . [ 4.405934] random: systemd: uninitialized urandom read (16 bytes read) [ 4.409124] systemd[1]: Initializing machine ID from random generator. [ 4.527140] random: ln: uninitialized urandom read (6 bytes read) [ 4.768736] random: systemd: uninitialized urandom read (16 bytes read) [ 4.770816] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.778540] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.784322] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.666766] device-mapper: uevent: version 1.0.3 [ 6.673302] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 8.644351] virtio_net virtio0 ens2: renamed from eth0 [ 8.663792] random: fast init done Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.170038] scsi host0: ata_piix [ 9.189886] scsi host1: ata_piix [ 9.193571] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 9.197237] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 14.873960] random: crng init done [ 14.878904] random: 7 urandom warning(s) missed due to ratelimiting [ 15.645564] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.792895] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.725107] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.117527] SELinux: Disabled at runtime. [ 19.186658] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.196429] systemd[1]: Detected virtualization kvm. [ 19.198525] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.412846] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.418210] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.427631] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.435477] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.441975] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.455135] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.469685] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 21.205196] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 23.048821] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.642366] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.792569] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.058498] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.324753] EDAC sbridge: Ver: 1.1.2 [ 28.033956] Key type dns_resolver registered [ 28.427746] NFS: Registering the id_resolver key type [ 28.429881] Key type id_resolver registered [ 28.431665] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg411-client login: [ 96.344968] libcfs: loading out-of-tree module taints kernel. [ 96.580410] Key type ._llcrypt registered [ 96.584403] Key type .llcrypt registered [ 97.742790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 97.763283] alg: No test for adler32 (adler32-zlib) [ 99.326281] Lustre: Lustre: Build Version: 2.17.50_3_g69d656b [ 100.010134] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 101.911180] Key type lgssc registered [ 103.736823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 194.361032] hrtimer: interrupt took 5025542 ns [ 255.182031] Lustre: Mounted lustre-client [ 261.456122] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 281.084087] Lustre: lustre-OST0000-osc-ffff89388cae5000: disconnect after 23s idle [ 285.461443] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing check_logdir /tmp/testlogs/ [ 292.741679] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing yml_node [ 298.126806] Lustre: DEBUG MARKER: Client: 2.17.50.3 [ 300.958121] Lustre: DEBUG MARKER: MDS: 2.17.50.3 [ 304.152338] Lustre: DEBUG MARKER: OSS: 2.17.50.3 [ 306.215820] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jan 22 04:20:10 EST 2026 [ 325.167941] LustreError: 7944:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff89388cae5000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 325.175708] LustreError: 7944:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 325.198332] Lustre: Unmounted lustre-client [ 354.039783] Key type lgssc unregistered [ 354.354957] LNet: 8391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 354.364121] LNetError: 8391:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 355.430943] LNet: Removed LNI 192.168.204.11@tcp [ 356.510290] Key type .llcrypt unregistered [ 356.513151] Key type ._llcrypt unregistered [ 389.389767] Key type ._llcrypt registered [ 389.392223] Key type .llcrypt registered [ 389.585679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 389.603801] alg: No test for adler32 (adler32-zlib) [ 390.664246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 390.734515] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 390.782795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 395.746254] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 395.904815] LNet: 9259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.912877] LNetError: 9259:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 395.938955] LNet: Removed LNI 192.168.204.11@tcp [ 396.712480] Key type .llcrypt unregistered [ 396.718519] Key type ._llcrypt unregistered [ 405.146674] Key type ._llcrypt registered [ 405.155932] Key type .llcrypt registered [ 407.150151] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 04:21:51 (1769073711) [ 407.588442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 407.600361] alg: No test for adler32 (adler32-zlib) [ 408.636181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.731075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.765527] LNet: 9676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.796410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.921491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 408.963297] LNet: 9683:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 417.710363] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 04:22:02 (1769073722) [ 418.087800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.147025] LNet: 10262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 418.196038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.334669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 427.891742] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 04:22:12 (1769073732) [ 428.208714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.236961] LNet: 10848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 428.256620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.373485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 437.892151] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 04:22:22 (1769073742) [ 438.118983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.181641] LNet: 11430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 438.227279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.390864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 447.125333] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 04:22:31 (1769073751) [ 447.436095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.478192] LNet: 12019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 447.517210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.651646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 448.504894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 458.164699] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 04:22:42 (1769073762) [ 458.404347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.444918] LNet: 12610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.509744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.641987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 459.419528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 460.455922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 470.565632] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 04:22:54 (1769073774) [ 470.713196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.793710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.921798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 481.030187] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 04:23:05 (1769073785) [ 481.201633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.249692] LNet: 13787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.269353] LNet: 13787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 481.301904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.422305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 481.464427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 481.923931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 481.973699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 482.753274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 482.838586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 483.688214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 483.760288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 484.607981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 484.682546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 485.585484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 485.639704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 486.482421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 486.570952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 487.363787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 487.434821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 488.308558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 488.421269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 489.407226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 489.476827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 490.063219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 490.104427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 490.758927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 490.875771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 491.502506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 491.575319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 501.531882] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 04:23:25 (1769073805) [ 501.636937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.723696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.821827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 501.882538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 511.222729] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 04:23:35 (1769073815) [ 511.381581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.442854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.625766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 511.671777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 519.920228] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 04:23:44 (1769073824) [ 520.151974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.190575] LNet: 15647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 520.201589] LNet: 15647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 520.274836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.391406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 520.422620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 530.270722] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 04:23:54 (1769073834) [ 530.412151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.516934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.648423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 530.674639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 539.894140] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 04:24:04 (1769073844) [ 540.179568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.235629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.334719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 540.402597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 549.216365] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 04:24:13 (1769073853) [ 549.334389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.445297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.616395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 549.707146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 558.835262] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 04:24:23 (1769073863) [ 559.019532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.107671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.245220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 560.003875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 568.721231] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 04:24:33 (1769073873) [ 568.865565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.950018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.239692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 569.605191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 577.799944] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 04:24:42 (1769073882) [ 577.923993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.970674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.206144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 578.541922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 586.494631] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 04:24:51 (1769073891) [ 586.919416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.936503] LNet: 19827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.940918] LNet: 19827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 586.960408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.711056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 588.133266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 597.651723] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 04:25:01 (1769073901) [ 597.798878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.909511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.055176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 598.111959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 606.356953] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 04:25:10 (1769073910) [ 606.514084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.565298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.732453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 606.822971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 615.338835] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 04:25:19 (1769073919) [ 615.506090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.582194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.684565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 624.098202] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 04:25:28 (1769073928) [ 624.273715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.354422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.551124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 632.799826] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 04:25:37 (1769073937) [ 632.939638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.989607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.111596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 633.482761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 642.527636] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 04:25:46 (1769073946) [ 642.653654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.741584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.914073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 642.948569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 651.250812] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 04:25:55 (1769073955) [ 651.410982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.471411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.649660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 651.709652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 660.147262] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 04:26:04 (1769073964) [ 660.478528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.555286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.702931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 660.761898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 667.446444] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 04:26:12 (1769073972) [ 667.853782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 667.936953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.051981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 668.128624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 676.042358] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 04:26:20 (1769073980) [ 676.346487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.383756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.439917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 676.527089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 676.554898] LNetError: 25802:0:(peer.c:2225:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 676.610820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 676.683645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 684.044956] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 04:26:28 (1769073988) [ 684.220558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.326118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.502467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MSOXKXNj2K [ 684.675794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MSOXKXNj2K [ 684.781646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MSOXKXNj2K [ 692.065833] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 04:26:36 (1769073996) [ 692.316399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.351871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.430133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 692.476500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 692.546604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 699.748396] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 04:26:44 (1769074004) [ 700.065230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.139642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.226413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 705.743485] LNet: Message forwarding will be enabled in 2s [ 707.807233] LNet: Message forwarding enabled [ 708.243877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 708.259181] LNet: Message forwarding will stop in 2s to 6s [ 713.077772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 714.705507] LNet: Message forwarding disabled [ 717.087904] LNet: Message forwarding will be enabled in 2s [ 719.839668] LNet: Message forwarding enabled [ 726.382936] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 04:27:10 (1769074030) [ 726.481233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.527246] LNet: 28233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.539593] LNet: 28233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 728.095847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.201168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 735.977822] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 04:27:20 (1769074040) [ 736.092336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.141805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.237679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 736.267851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 736.326901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 736.369054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 736.420133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 736.462926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 736.495628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 736.529571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 736.569607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 736.613679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 736.640848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 736.705825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 736.780625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 736.827760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 736.883298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 736.938980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 736.987264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 737.025665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 737.052845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 737.077660] LNetError: 28866:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 744.429289] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 04:27:29 (1769074049) [ 745.045159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.089966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.174966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 752.126405] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 04:27:37 (1769074057) [ 754.298536] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg411-server [ 756.271198] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 04:27:40 (1769074060) [ 756.531626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 756.566209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.611563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.628727] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 758.514839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[12-16]@tcp [ 758.565093] LNet: 30585:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 758.645587] LNetError: 30375:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.15@tcp is being used as a gateway but routing feature is not turned on [ 758.676044] LNetError: 30375:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.15@tcp has gone from up to down [ 759.656710] LNetError: 30375:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 759.687841] LNetError: 30375:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 760.707603] LNetError: 30375:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.14@tcp is being used as a gateway but routing feature is not turned on [ 760.729540] LNetError: 30375:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 762.733476] LNetError: 30375:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.14@tcp is being used as a gateway but routing feature is not turned on [ 762.753617] LNetError: 30375:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 766.433706] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 04:27:50 (1769074070) [ 766.820092] LNetError: 30375:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 766.843072] LNetError: 30375:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 768.769419] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg411-server [ 770.734708] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 04:27:55 (1769074075) [ 770.813974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.854340] LNetError: 31514:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.884091] LNet: Removed LNI 192.168.204.11@tcp [ 770.945434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.983586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 771.000035] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 771.647208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[12-17/2]@tcp [ 771.680971] LNet: 31637:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 771.694103] LNet: 31637:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 771.726925] LNetError: 31524:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.12@tcp has gone from up to down [ 771.733662] LNetError: 31524:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 771.740413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[12-17/2]@tcp [ 778.789651] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 04:28:03 (1769074083) [ 779.043776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.070692] LNetError: 32229:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.137115] LNet: Removed LNI 192.168.204.11@tcp [ 780.209119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.323268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 780.420141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.462538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.543203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 780.612798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.695149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.826872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 780.995818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.073649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.207312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 781.333707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.381367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.454334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 781.553253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.608926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.713070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 781.866708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.929830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.039758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 782.137436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 782.225209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.336439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 782.432187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 782.482602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.544545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 782.656406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 782.714577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.810850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 782.942381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 782.992309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.077253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 783.153882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.184211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.249912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 790.481110] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 04:28:15 (1769074095) [ 792.501509] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg411-server [ 794.590641] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 04:28:19 (1769074099) [ 796.460470] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg411-server [ 798.654526] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 04:28:23 (1769074103) [ 799.513261] Key type .llcrypt unregistered [ 799.514395] Key type ._llcrypt unregistered [ 807.005753] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 807.276063] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 807.289740] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 807.310436] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 810.150920] Key type ._llcrypt registered [ 810.153781] Key type .llcrypt registered [ 810.450891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.474964] alg: No test for adler32 (adler32-zlib) [ 811.425039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.514785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.540768] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 812.462546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 812.501134] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 815.485427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 815.519524] LNet: Removed LNI 10.1.2.3@tcp [ 815.709831] LNet: 34349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.739336] LNetError: 34349:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.775538] LNet: Removed LNI 192.168.204.11@tcp [ 816.469805] Key type .llcrypt unregistered [ 816.477907] Key type ._llcrypt unregistered [ 832.906593] Key type ._llcrypt registered [ 832.914055] Key type .llcrypt registered [ 834.677492] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 04:28:58 (1769074138) [ 835.358583] Key type .llcrypt unregistered [ 835.361371] Key type ._llcrypt unregistered [ 842.500665] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 842.793180] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 842.809612] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 842.812427] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 845.712956] Key type ._llcrypt registered [ 845.715582] Key type .llcrypt registered [ 846.010361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 846.030139] alg: No test for adler32 (adler32-zlib) [ 847.046514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.109229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 847.161302] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 848.149089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 848.176666] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 851.351356] LNet: 35837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.362773] LNetError: 35837:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.381410] LNet: Removed LNI 192.168.204.11@tcp [ 851.405605] LNet: Removed LNI 10.1.2.3@tcp [ 852.117518] Key type .llcrypt unregistered [ 852.121617] Key type ._llcrypt unregistered [ 866.233580] Key type ._llcrypt registered [ 866.236557] Key type .llcrypt registered [ 868.451745] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 04:29:32 (1769074172) [ 869.182571] Key type .llcrypt unregistered [ 869.186553] Key type ._llcrypt unregistered [ 878.239913] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 878.490525] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 878.492981] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 878.514338] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 881.394853] Key type ._llcrypt registered [ 881.396195] Key type .llcrypt registered [ 881.708237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 881.742087] alg: No test for adler32 (adler32-zlib) [ 882.622685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.665917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 882.682490] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 885.209457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 885.252316] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 888.073798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 888.098109] LNet: Removed LNI 10.1.2.3@tcp [ 888.139753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 888.198472] LNet: Removed LNI 10.1.2.31@tcp [ 888.280656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 888.339665] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 889.617379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 889.662954] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 891.164205] LNet: 37753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.175594] LNetError: 37753:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 891.187684] LNet: Removed LNI 10.1.2.3@tcp [ 891.204117] LNet: Removed LNI 10.1.2.31@tcp [ 891.816519] Key type .llcrypt unregistered [ 891.826151] Key type ._llcrypt unregistered [ 909.633905] Key type ._llcrypt registered [ 909.638345] Key type .llcrypt registered [ 911.206399] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 04:30:15 (1769074215) [ 911.977346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 912.036436] alg: No test for adler32 (adler32-zlib) [ 913.146625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.239898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 913.287503] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 914.409794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.447825] LNet: 38705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.468582] LNetError: 38705:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.496184] LNet: Removed LNI 192.168.204.11@tcp [ 914.523787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.615874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.11@tcp [ 914.659323] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 915.748631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.11@tcp [ 915.809858] LNet: Removed LNI 192.168.204.11@tcp [ 915.920550] LNet: 38840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.601449] Key type .llcrypt unregistered [ 916.602699] Key type ._llcrypt unregistered [ 928.880517] Key type ._llcrypt registered [ 928.887200] Key type .llcrypt registered [ 930.503507] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 04:30:34 (1769074234) [ 932.653528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg411-server [ 934.892283] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 04:30:39 (1769074239) [ 935.462408] Key type .llcrypt unregistered [ 935.463773] Key type ._llcrypt unregistered [ 942.938534] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 943.252357] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 943.289736] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 943.296742] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 946.052696] Key type ._llcrypt registered [ 946.055773] Key type .llcrypt registered [ 946.392917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 946.430276] alg: No test for adler32 (adler32-zlib) [ 947.404827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.486334] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 948.527520] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 951.273718] LNet: 40671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.280802] LNetError: 40671:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.286896] LNet: Removed LNI 192.168.204.11@tcp1 [ 951.301381] LNet: Removed LNI 10.1.2.3@tcp2 [ 952.197620] Key type .llcrypt unregistered [ 952.203317] Key type ._llcrypt unregistered [ 967.027836] Key type ._llcrypt registered [ 967.030716] Key type .llcrypt registered [ 968.451287] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 04:31:13 (1769074273) [ 969.044722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 969.069359] alg: No test for adler32 (adler32-zlib) [ 970.044899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.111937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 970.132318] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 971.179708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.214333] LNet: 41637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.232750] LNetError: 41637:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.260070] LNet: Removed LNI 192.168.204.11@tcp [ 971.291854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.434526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 971.470662] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 979.394557] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 04:31:23 (1769074283) [ 979.509344] LNet: 42223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.521462] LNetError: 42223:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.540334] LNet: Removed LNI 192.168.204.11@tcp [ 980.111069] Key type .llcrypt unregistered [ 980.113307] Key type ._llcrypt unregistered [ 986.783778] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 987.099316] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 987.107384] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 987.115731] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 989.855524] Key type ._llcrypt registered [ 989.858906] Key type .llcrypt registered [ 990.145685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 990.183681] alg: No test for adler32 (adler32-zlib) [ 991.217824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.306709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 991.335695] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 992.587555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 992.627533] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 994.859475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 994.886793] LNet: 42972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.898932] LNetError: 42972:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.906550] LNet: Removed LNI 192.168.204.11@tcp [ 994.932902] LNet: Removed LNI 10.1.2.3@tcp [ 994.979224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.102731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 995.134354] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 995.154097] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 996.005886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 996.022266] LNet: 42999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.028431] LNetError: 42999:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 996.033846] LNetError: 42999:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 996.039485] LNet: Removed LNI 192.168.204.11@tcp [ 996.048600] LNet: Removed LNI 10.1.2.3@tcp [ 996.073387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 996.142407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 996.166531] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 996.173731] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 996.694380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 996.719733] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 998.333317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.352238] LNet: 43161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.355586] LNetError: 43161:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.359868] LNetError: 43161:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 998.365452] LNet: Removed LNI 192.168.204.11@tcp [ 998.367751] LNet: Removed LNI 10.1.2.3@tcp [ 998.377340] LNet: Removed LNI 192.168.204.11@tcp2 [ 998.407370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.473453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 998.487742] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 998.490830] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 998.494299] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1006.370184] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 04:31:51 (1769074311) [ 1006.495354] LNet: 43750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.501795] LNetError: 43750:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.510173] LNetError: 43750:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1006.518635] LNet: Removed LNI 192.168.204.11@tcp [ 1006.521747] LNet: Removed LNI 10.1.2.3@tcp [ 1006.560916] LNet: Removed LNI 192.168.204.11@tcp2 [ 1007.292917] Key type .llcrypt unregistered [ 1007.294535] Key type ._llcrypt unregistered [ 1014.511751] Key type ._llcrypt registered [ 1014.515524] Key type .llcrypt registered [ 1014.663904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1014.676491] alg: No test for adler32 (adler32-zlib) [ 1015.795610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.924420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1015.970858] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1017.306741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1017.321120] LNet: 44149:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.324994] LNetError: 44149:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.337777] LNet: Removed LNI 192.168.204.11@tcp [ 1017.356684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1017.444454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1017.464720] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1018.101559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1018.134210] LNet: 44179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.139647] LNetError: 44179:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.151725] LNet: Removed LNI 192.168.204.11@tcp [ 1018.169797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.267840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1018.299715] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1019.000635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1019.040124] LNet: Removed LNI 192.168.204.11@tcp [ 1019.112960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1019.264559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1019.315612] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1019.816019] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1020.109674] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1020.117836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1020.153423] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1022.540412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1022.562188] LNet: 44289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.570578] LNet: 44289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1022.579161] LNetError: 44289:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.591224] LNetError: 44289:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1022.635745] LNet: Removed LNI 192.168.204.11@tcp [ 1022.656848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1022.744521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1022.805538] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1033.523700] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 04:32:17 (1769074337) [ 1033.847277] LNet: 44887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.861797] LNetError: 44887:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1033.888682] LNet: Removed LNI 192.168.204.11@tcp [ 1034.732696] Key type .llcrypt unregistered [ 1034.738713] Key type ._llcrypt unregistered [ 1041.489365] Key type ._llcrypt registered [ 1041.491806] Key type .llcrypt registered [ 1041.785126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1041.817513] alg: No test for adler32 (adler32-zlib) [ 1042.763806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.833964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1042.852500] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1043.800206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.823026] LNet: 45283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.832690] LNetError: 45283:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.843890] LNet: Removed LNI 192.168.204.11@tcp [ 1043.867415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.956877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1043.985670] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1053.120192] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 04:32:37 (1769074357) [ 1053.275326] LNet: 45868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.281971] LNetError: 45868:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1053.322546] LNet: Removed LNI 192.168.204.11@tcp [ 1054.136675] Key type .llcrypt unregistered [ 1054.141762] Key type ._llcrypt unregistered [ 1062.790961] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1063.233127] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1063.246712] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1063.248630] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1066.135414] Key type ._llcrypt registered [ 1066.140478] Key type .llcrypt registered [ 1066.513158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1066.526205] alg: No test for adler32 (adler32-zlib) [ 1067.575891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1067.659442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1067.702700] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1068.979189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1069.032802] LNet: 46323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.048049] LNetError: 46323:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1069.066321] LNet: Removed LNI 192.168.204.11@tcp [ 1069.087104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1069.177587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1069.221836] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1077.862886] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 04:33:02 (1769074382) [ 1077.972162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1078.022341] LNet: 46917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1078.031064] LNetError: 46917:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1078.068874] LNet: Removed LNI 192.168.204.11@tcp [ 1078.148965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1078.314387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1084.933963] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 04:33:10 (1769074390) [ 1085.283378] LNet: 47505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.955868] Key type .llcrypt unregistered [ 1085.957453] Key type ._llcrypt unregistered [ 1094.193665] Key type ._llcrypt registered [ 1094.196987] Key type .llcrypt registered [ 1094.533957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1094.592196] alg: No test for adler32 (adler32-zlib) [ 1095.543497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1095.628938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1095.712977] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1096.764233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1096.783938] LNet: 47899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.788762] LNetError: 47899:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1096.812959] LNet: Removed LNI 192.168.204.11@tcp [ 1096.846396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1096.927479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1096.946532] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1104.728537] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 04:33:29 (1769074409) [ 1105.101805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1105.135804] LNet: 48494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1105.155766] LNetError: 48494:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1105.184598] LNet: Removed LNI 192.168.204.11@tcp [ 1105.238225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1105.298939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1112.871952] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 04:33:37 (1769074417) [ 1114.686572] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1116.895413] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 04:33:41 (1769074421) [ 1118.815198] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1120.639259] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 04:33:45 (1769074425) [ 1122.760495] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1124.859727] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 04:33:49 (1769074429) [ 1126.848522] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1129.325168] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 04:33:53 (1769074433) [ 1131.359060] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1133.434279] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 04:33:58 (1769074438) [ 1133.653519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1133.718902] LNet: 50544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.786671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1133.864258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1133.908815] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1134.686492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.11 [ 1134.708778] LNet: 50680:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1134.720845] LNet: Removed LNI 192.168.204.11@tcp [ 1141.259670] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 04:34:06 (1769074446) [ 1141.516456] LNet: 51251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1141.524845] LNetError: 51251:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1141.542040] LNet: Removed LNI 192.168.204.11@tcp [ 1142.276777] Key type .llcrypt unregistered [ 1142.278631] Key type ._llcrypt unregistered [ 1148.851976] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1149.191779] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1149.201462] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1149.203586] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1151.992462] Key type ._llcrypt registered [ 1151.995922] Key type .llcrypt registered [ 1152.176323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1152.188722] alg: No test for adler32 (adler32-zlib) [ 1153.674321] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1161.444662] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 04:34:25 (1769074465) [ 1161.561436] LNet: 52145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.577527] LNetError: 52145:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1161.599992] LNet: Removed LNI 10.1.2.3@tcp [ 1162.177945] Key type .llcrypt unregistered [ 1162.180153] Key type ._llcrypt unregistered [ 1169.733890] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1170.043905] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1170.054357] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1170.072537] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1173.158253] Key type ._llcrypt registered [ 1173.161275] Key type .llcrypt registered [ 1173.412997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1173.439631] alg: No test for adler32 (adler32-zlib) [ 1174.535772] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1182.436781] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 04:34:46 (1769074486) [ 1182.626145] LNet: 53035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.640708] LNetError: 53035:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1182.695142] LNet: Removed LNI 10.1.2.3@tcp [ 1183.359601] Key type .llcrypt unregistered [ 1183.365533] Key type ._llcrypt unregistered [ 1191.299784] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1191.622949] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1191.629001] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1191.638052] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1193.412700] Key type ._llcrypt registered [ 1193.415176] Key type .llcrypt registered [ 1193.606936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1193.623811] alg: No test for adler32 (adler32-zlib) [ 1194.651797] LNet: 53332:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1194.678493] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1202.514778] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 04:35:07 (1769074507) [ 1202.773135] LNet: 53922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.781868] LNetError: 53922:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1202.827359] LNet: Removed LNI 10.1.2.3@tcp [ 1203.665569] Key type .llcrypt unregistered [ 1203.667987] Key type ._llcrypt unregistered [ 1211.079513] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1211.404451] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1211.416702] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1211.428265] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1214.280874] Key type ._llcrypt registered [ 1214.282092] Key type .llcrypt registered [ 1214.583683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1214.607543] alg: No test for adler32 (adler32-zlib) [ 1215.540416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1215.629177] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1225.969395] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 04:35:30 (1769074530) [ 1226.750303] LNet: 55153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.773192] LNetError: 55153:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1226.802213] LNet: Removed LNI 10.1.2.3@tcp [ 1227.890699] Key type .llcrypt unregistered [ 1227.893533] Key type ._llcrypt unregistered [ 1234.817119] Key type ._llcrypt registered [ 1234.824732] Key type .llcrypt registered [ 1235.085042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1235.106976] alg: No test for adler32 (adler32-zlib) [ 1236.088716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1236.142506] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1245.273631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1246.070367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_interrupt -r 1 [ 1246.105446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_interrupt -r 1 [ 1246.141539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1246.186067] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1246.372792] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1246.472541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1246.503455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1246.527537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1246.555235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1246.582953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1246.608249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1246.637673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1246.663974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1246.679326] LNet: 55587:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1246.698196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1246.724156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1246.766505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1246.816285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1246.858594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1246.889569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1246.921332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1247.401411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_dropped -r 1 [ 1247.430995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_dropped -r 1 [ 1247.459604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1247.477827] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1247.683623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1247.717964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1247.744606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1247.767610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1247.794024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1247.818605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.854684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.890923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.909039] LNet: 55697:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.930942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.963277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1248.003703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1248.062580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1248.098520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.142472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1248.189202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1248.801650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_aborted -r 1 [ 1248.850545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_aborted -r 1 [ 1248.906675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1248.948491] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1249.269538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1249.302532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1249.341261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.372571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.400845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.437374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.503351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.578226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.605318] LNet: 55806:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1249.629644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.664526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.713129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.751482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.800474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.860439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1249.912954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1250.628271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_no_route -r 1 [ 1250.676921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_no_route -r 1 [ 1250.766335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1251.032566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1251.066823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1251.125021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1251.171712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1251.247490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1251.304563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1251.354792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1251.408920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1251.493719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1251.567610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1251.627194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1251.687967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1251.755423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1251.809780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1251.862849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1252.539671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_timeout -r 1 [ 1252.585155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_timeout -r 1 [ 1252.632817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1252.649822] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1252.656543] LNet: Skipped 2 previous similar messages [ 1252.922583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1252.950114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1252.978439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1253.004911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1253.035776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1253.057885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1253.081524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1253.108981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1253.128930] LNet: 56025:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1253.134594] LNet: 56025:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1253.154621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1253.200880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1253.236323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1253.264456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1253.328048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1253.393516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1253.459911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1254.197557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_error -r 1 [ 1254.250369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_error -r 1 [ 1254.282905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1254.524457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1254.578388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1254.645967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1254.687986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1254.767851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1254.813424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1254.873986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1254.926366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1254.966813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1255.023559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1255.065669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1255.114563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1255.172388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1255.227106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1255.288195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1255.671357] LNet: 56174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.681157] LNetError: 56174:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1256.742943] LNet: Removed LNI 192.168.204.11@tcp [ 1257.434128] Key type .llcrypt unregistered [ 1257.441979] Key type ._llcrypt unregistered [ 1279.099735] Key type ._llcrypt registered [ 1279.101886] Key type .llcrypt registered [ 1280.634184] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 04:36:25 (1769074585) [ 1281.511377] Key type .llcrypt unregistered [ 1281.512908] Key type ._llcrypt unregistered [ 1289.197373] Key type ._llcrypt registered [ 1289.202569] Key type .llcrypt registered [ 1289.542289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1289.575797] alg: No test for adler32 (adler32-zlib) [ 1290.524835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1290.551803] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1299.265539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1311.817466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1311.880072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1311.912270] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1313.956548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_interrupt -r 1 [ 1313.997840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_interrupt -r 1 [ 1314.065536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_interrupt -r 1 [ 1314.150449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_interrupt -r 1 [ 1314.230155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_interrupt -r 1 [ 1314.327028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_interrupt -r 1 [ 1314.391898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_interrupt -r 1 [ 1314.435449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_interrupt -r 1 [ 1314.477147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1314.505725] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1314.511515] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1314.786923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1314.817675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1314.872028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1314.908347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1314.958720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1315.014302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1315.063613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1315.118693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1315.141674] LNet: 57643:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1315.162905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1315.206296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1315.243098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1315.278360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1315.318434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1315.365388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1315.427689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1316.114602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_dropped -r 1 [ 1316.141755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_dropped -r 1 [ 1316.171485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_dropped -r 1 [ 1316.211954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_dropped -r 1 [ 1316.247417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_dropped -r 1 [ 1316.276698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_dropped -r 1 [ 1316.325232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_dropped -r 1 [ 1316.381408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_dropped -r 1 [ 1316.425965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1316.440467] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1316.444979] LNet: Skipped 2 previous similar messages [ 1316.660063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1316.690932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1316.717815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1316.743177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1316.775533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1316.802216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1316.833842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1316.856185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1316.873775] LNet: 57764:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1316.889978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1316.913891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1316.942541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1316.977916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1317.021476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1317.066864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1317.101869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1317.609954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_aborted -r 1 [ 1317.655883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_aborted -r 1 [ 1317.695680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_aborted -r 1 [ 1317.728964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_aborted -r 1 [ 1317.769510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_aborted -r 1 [ 1317.802873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_aborted -r 1 [ 1317.856703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_aborted -r 1 [ 1317.913542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_aborted -r 1 [ 1317.950494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1317.971184] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1317.978349] LNet: Skipped 2 previous similar messages [ 1318.206317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1318.242127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1318.279657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1318.312737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1318.345159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1318.381841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1318.446976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1318.496432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1318.535716] LNet: 57886:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1318.563593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1318.620743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1318.687626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1318.737095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1318.778399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1318.818724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1318.863746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1319.500815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_no_route -r 1 [ 1319.565747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_no_route -r 1 [ 1319.627682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_no_route -r 1 [ 1319.687446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_no_route -r 1 [ 1319.767372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_no_route -r 1 [ 1319.806565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_no_route -r 1 [ 1319.837913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_no_route -r 1 [ 1319.885563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_no_route -r 1 [ 1319.954929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1319.996454] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1320.005297] LNet: Skipped 2 previous similar messages [ 1320.275414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1320.299881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1320.331378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1320.366461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1320.405918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1320.446243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1320.478951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1320.509995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1320.539232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1320.603364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1320.668961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1320.720701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1320.777772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1320.848933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1320.886732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1321.569726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_timeout -r 1 [ 1321.613093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_timeout -r 1 [ 1321.678286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_timeout -r 1 [ 1321.712917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_timeout -r 1 [ 1321.769611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_timeout -r 1 [ 1321.809908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_timeout -r 1 [ 1321.877625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_timeout -r 1 [ 1321.925943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_timeout -r 1 [ 1321.969478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1322.333583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1322.381685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1322.441898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1322.502494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1322.544659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1322.592029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1322.667093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1322.699874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1322.740068] LNet: 58128:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1322.753354] LNet: 58128:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1322.785248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1322.840857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1322.889835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1322.926603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1322.965304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1322.996783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1323.027973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1323.560328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_error -r 1 [ 1323.595695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_error -r 1 [ 1323.666602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_error -r 1 [ 1323.724265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_error -r 1 [ 1323.775116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_error -r 1 [ 1323.818666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_error -r 1 [ 1323.861931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_error -r 1 [ 1323.896664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_error -r 1 [ 1323.940664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1324.075004] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1324.087698] LNet: Skipped 7 previous similar messages [ 1324.265210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1324.303616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1324.338974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1324.386154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1324.440693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1324.483175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1324.521393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1324.573938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1324.606456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1324.646512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1324.675694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1324.703996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1324.738268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1324.781959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1324.823293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1331.419907] LNet: 58325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1331.439282] LNetError: 58325:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1331.450817] LNet: Removed LNI 192.168.204.11@tcp [ 1332.527854] LNet: Removed LNI 192.168.204.11@tcp1 [ 1333.030260] Key type .llcrypt unregistered [ 1333.033690] Key type ._llcrypt unregistered [ 1355.086659] Key type ._llcrypt registered [ 1355.088375] Key type .llcrypt registered [ 1356.985992] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 04:37:40 (1769074660) [ 1357.966511] Key type .llcrypt unregistered [ 1357.972083] Key type ._llcrypt unregistered [ 1368.231475] Key type ._llcrypt registered [ 1368.233046] Key type .llcrypt registered [ 1368.758522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1368.826685] alg: No test for adler32 (adler32-zlib) [ 1370.003401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1370.122856] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1380.965529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1382.001785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_dropped -r 1 [ 1382.027115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_dropped -r 1 [ 1382.044668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1382.070713] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1382.414452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1382.486221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1382.560545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1382.637605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1382.726145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1382.808790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1382.868710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1382.934959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1382.963274] LNet: 59571:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1383.000833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1383.050827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1383.092822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1383.143923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1383.191944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1383.262423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1383.331413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1383.996769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_error -r 1 [ 1384.053764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_error -r 1 [ 1384.125531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1384.179746] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1384.556566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1384.587231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1384.653907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1384.688792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1384.724229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1384.770577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1384.815377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1384.880901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1384.904398] LNet: 59680:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1384.937929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1384.985466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1385.072217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1385.136203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1385.204845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1385.257295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1385.285095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1385.849981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_timeout -r 1 [ 1385.904768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_timeout -r 1 [ 1385.992807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1386.016852] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1386.394413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1386.435680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1386.481478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1386.517443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1386.554549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1386.629364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1386.678826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1386.762068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1386.790966] LNet: 59791:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1386.839306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1386.885324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1386.933147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1386.986334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1387.031824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1387.101898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1387.168249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1387.491180] LNet: 59831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.508577] LNetError: 59831:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.530315] LNet: Removed LNI 192.168.204.11@tcp [ 1388.147542] Key type .llcrypt unregistered [ 1388.152950] Key type ._llcrypt unregistered [ 1407.238609] Key type ._llcrypt registered [ 1407.240453] Key type .llcrypt registered [ 1408.838601] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 04:38:33 (1769074713) [ 1409.666435] Key type .llcrypt unregistered [ 1409.668412] Key type ._llcrypt unregistered [ 1416.761660] Key type ._llcrypt registered [ 1416.763462] Key type .llcrypt registered [ 1417.032935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1417.050239] alg: No test for adler32 (adler32-zlib) [ 1418.034719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1418.073901] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1427.486343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1437.413503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1437.480708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1437.530017] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1439.367810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_dropped -r 1 [ 1439.395493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e remote_dropped -r 1 [ 1439.422179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_dropped -r 1 [ 1439.447444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e remote_dropped -r 1 [ 1439.473872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e remote_dropped -r 1 [ 1439.504804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e remote_dropped -r 1 [ 1439.536474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e remote_dropped -r 1 [ 1439.579647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e remote_dropped -r 1 [ 1439.626480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1439.655738] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1439.662849] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.111@tcp [ 1439.917353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1439.950718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1439.976968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1440.002654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1440.027992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1440.059052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1440.096439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1440.125478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1440.139375] LNet: 61292:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1440.153971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1440.181460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1440.210955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1440.235975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1440.258822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1440.286391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1440.316922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1440.339145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1440.803207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_error -r 1 [ 1440.839245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e remote_error -r 1 [ 1440.876603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_error -r 1 [ 1440.905832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e remote_error -r 1 [ 1440.945364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e remote_error -r 1 [ 1440.997841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e remote_error -r 1 [ 1441.039316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e remote_error -r 1 [ 1441.091098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e remote_error -r 1 [ 1441.141916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1441.162718] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1441.172370] LNet: Skipped 2 previous similar messages [ 1441.446677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1441.484433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1441.519707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1441.561934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1441.611257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1441.655733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1441.710313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1441.755324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1441.778461] LNet: 61415:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1441.797639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1441.824940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1441.855691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1441.882951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1441.929713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1441.965279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1441.996779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1442.029232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1442.553923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_timeout -r 1 [ 1442.604209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e remote_timeout -r 1 [ 1442.675115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_timeout -r 1 [ 1442.730972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e remote_timeout -r 1 [ 1442.768201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e remote_timeout -r 1 [ 1442.796915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e remote_timeout -r 1 [ 1442.844609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e remote_timeout -r 1 [ 1442.914820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e remote_timeout -r 1 [ 1442.980970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1443.020072] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1443.300588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1443.331192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1443.378093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1443.437881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1443.523603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1443.579890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1443.609876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1443.641155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1443.668115] LNet: 61539:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1443.689426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1443.765890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1443.879512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1443.958122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1444.034992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1444.080394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1444.145290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1444.189248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1449.351202] LNet: 61615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.356670] LNetError: 61615:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.399152] LNet: Removed LNI 192.168.204.11@tcp [ 1450.406067] LNetError: 61615:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.431369] LNet: Removed LNI 192.168.204.11@tcp1 [ 1451.067553] Key type .llcrypt unregistered [ 1451.080201] Key type ._llcrypt unregistered [ 1470.669951] Key type ._llcrypt registered [ 1470.677698] Key type .llcrypt registered [ 1472.174833] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 04:39:36 (1769074776) [ 1472.641447] Key type .llcrypt unregistered [ 1472.643548] Key type ._llcrypt unregistered [ 1480.068059] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1480.273538] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1480.284187] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1480.294575] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1481.925799] Key type ._llcrypt registered [ 1481.927819] Key type .llcrypt registered [ 1482.049964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1482.062814] alg: No test for adler32 (adler32-zlib) [ 1483.074965] LNet: 62680:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1483.093146] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1483.153089] LNet: 62694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1483.158451] LNetError: 62694:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1483.170972] LNet: Removed LNI 192.168.204.11@tcp [ 1483.955372] Key type .llcrypt unregistered [ 1483.961275] Key type ._llcrypt unregistered [ 1490.643706] Key type ._llcrypt registered [ 1490.646025] Key type .llcrypt registered [ 1490.852557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1490.870874] alg: No test for adler32 (adler32-zlib) [ 1491.893942] LNet: 62950:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1491.932141] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1491.943445] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1492.037481] LNet: 62964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.050476] LNetError: 62964:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.058555] LNet: Removed LNI 192.168.204.11@tcp [ 1492.083576] LNet: Removed LNI 10.1.2.3@tcp1 [ 1492.779873] Key type .llcrypt unregistered [ 1492.783458] Key type ._llcrypt unregistered [ 1500.967474] Key type ._llcrypt registered [ 1500.976453] Key type .llcrypt registered [ 1501.228436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1501.253858] alg: No test for adler32 (adler32-zlib) [ 1502.246605] LNet: 63219:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1502.254825] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1502.258574] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1502.299681] LNet: 63233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1502.306403] LNetError: 63233:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1502.313983] LNet: Removed LNI 192.168.204.11@tcp [ 1502.329900] LNet: Removed LNI 10.1.2.3@tcp [ 1503.373429] Key type .llcrypt unregistered [ 1503.376224] Key type ._llcrypt unregistered [ 1511.191609] Key type ._llcrypt registered [ 1511.195039] Key type .llcrypt registered [ 1511.562603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1511.606083] alg: No test for adler32 (adler32-zlib) [ 1512.704914] LNet: 63490:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1512.719985] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1512.724618] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1512.794944] LNet: 63504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1512.808569] LNetError: 63504:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1512.825051] LNet: Removed LNI 192.168.204.11@tcp [ 1512.837968] LNet: Removed LNI 10.1.2.3@tcp [ 1513.621803] Key type .llcrypt unregistered [ 1513.634266] Key type ._llcrypt unregistered [ 1522.143870] Key type ._llcrypt registered [ 1522.158395] Key type .llcrypt registered [ 1522.619590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1522.675633] alg: No test for adler32 (adler32-zlib) [ 1523.713400] LNet: 63765:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1523.733113] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1523.739640] LNetError: 63765:0:(config.c:1560:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1523.748699] LNetError: Error -100 starting up LNI tcp [ 1523.763501] LNet: Removed LNI 192.168.204.11@tcp [ 1524.633774] Key type .llcrypt unregistered [ 1524.636685] Key type ._llcrypt unregistered [ 1537.425778] Key type ._llcrypt registered [ 1537.427406] Key type .llcrypt registered [ 1538.780563] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 04:40:43 (1769074843) [ 1539.939472] Key type .llcrypt unregistered [ 1539.940954] Key type ._llcrypt unregistered [ 1547.535910] Key type ._llcrypt registered [ 1547.538400] Key type .llcrypt registered [ 1547.862117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1547.886506] alg: No test for adler32 (adler32-zlib) [ 1548.906895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1548.962989] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1558.511718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1559.318139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e network_timeout -r 1 [ 1559.361321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e network_timeout -r 1 [ 1559.416737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1559.447989] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1559.731773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1559.794341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1559.831161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1559.893127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1559.958028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1560.053299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1560.129944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1560.168650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1560.202314] LNet: 65016:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1560.223419] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1560.248259] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1560.255193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1560.301209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1560.344332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1560.387088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1560.450359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1560.486385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1560.759879] LNet: 65054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.766067] LNetError: 65054:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1561.833514] LNet: Removed LNI 192.168.204.11@tcp [ 1562.320454] Key type .llcrypt unregistered [ 1562.322654] Key type ._llcrypt unregistered [ 1583.249117] Key type ._llcrypt registered [ 1583.251338] Key type .llcrypt registered [ 1583.393916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1583.403608] alg: No test for adler32 (adler32-zlib) [ 1584.360259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1584.403289] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1592.958070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1602.577244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1602.627624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1602.674657] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1604.091995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e network_timeout -r 1 [ 1604.126283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e network_timeout -r 1 [ 1604.179610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e network_timeout -r 1 [ 1604.215298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e network_timeout -r 1 [ 1604.253771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e network_timeout -r 1 [ 1604.293608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e network_timeout -r 1 [ 1604.334643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e network_timeout -r 1 [ 1604.387103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e network_timeout -r 1 [ 1604.448906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1604.475965] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1604.707955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1604.745695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1604.778995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1604.823970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1604.877115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1604.895389] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1604.902850] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.111@tcp [ 1604.932747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1604.977545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1605.023604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1605.045905] LNet: 65928:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1605.067794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1605.107575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1605.138816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1605.172969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1605.211693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1605.254303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1610.925657] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1611.326331] LNet: 66001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.332231] LNetError: 66001:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1611.341590] LNet: Removed LNI 192.168.204.11@tcp [ 1611.354150] LNet: Removed LNI 192.168.204.11@tcp1 [ 1611.793588] Key type .llcrypt unregistered [ 1611.797838] Key type ._llcrypt unregistered [ 1630.462445] Key type ._llcrypt registered [ 1630.481684] Key type .llcrypt registered [ 1632.394907] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 04:42:16 (1769074936) [ 1632.942973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1632.964575] alg: No test for adler32 (adler32-zlib) [ 1633.919465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.958366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1633.989079] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1634.793740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1634.808633] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1635.559774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1635.639162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1635.685728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1635.726854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1635.784912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1635.856861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1635.910883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.11@tcp [ 1635.937586] LNetError: 67098:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.11@tcp to recovery [ 1636.000034] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1636.000794] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1637.023470] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1637.417857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1641.121256] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1642.179268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1642.238493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1642.352739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.380439] LNet: 67195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1642.383796] LNetError: 67195:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1642.395795] LNet: Removed LNI 192.168.204.11@tcp [ 1642.408773] LNet: Removed LNI 192.168.204.11@tcp1 [ 1642.427853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.479825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1642.511901] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1643.314362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1643.347676] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1644.424901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1644.516587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1644.602139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1644.681980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1644.729338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1644.776928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1644.838964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1644.879160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.11@tcp [ 1644.911540] LNetError: 67467:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.11@tcp to recovery [ 1645.536699] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1648.545916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1651.679806] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1651.690404] LNet: Skipped 1 previous similar message [ 1656.651983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1656.691582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1656.730946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1662.246434] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 04:42:47 (1769074967) [ 1662.296783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.307993] LNet: 68178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.311498] LNetError: 68178:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.314584] LNetError: 68178:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1662.320095] LNet: Removed LNI 192.168.204.11@tcp [ 1662.329805] LNet: Removed LNI 192.168.204.11@tcp1 [ 1662.343120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.370396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1662.383533] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1662.738317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1662.749299] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1663.115781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1663.148910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1663.165526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1663.185015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1663.203048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1663.220868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1663.240755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.11@tcp --health 0 [ 1663.284452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1663.391158] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.11@tcp [ 1664.415389] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1664.418342] LNet: Skipped 1 previous similar message [ 1679.700729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1679.743996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1679.755233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1679.765208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.11@tcp --health 500 [ 1679.795686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1685.958777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1685.981534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.991560] LNet: 68691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.995693] LNetError: 68691:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.999795] LNetError: 68691:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1686.003853] LNet: Removed LNI 192.168.204.11@tcp [ 1686.012326] LNet: Removed LNI 192.168.204.11@tcp1 [ 1686.019738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.038206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1686.048765] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1686.293508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1686.301434] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1686.570522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1686.582958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1686.603733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1686.614881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1686.627085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1686.638981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1686.650813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1686.661929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.11@tcp --health 0 [ 1687.071318] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1687.074521] LNet: Skipped 3 previous similar messages [ 1687.759947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1698.146343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1698.158242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1698.170133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1700.434452] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 04:43:26 (1769075006) [ 1700.583770] LNet: 69747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.587467] LNetError: 69747:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.590884] LNetError: 69747:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1700.594502] LNet: Removed LNI 192.168.204.11@tcp [ 1700.599583] LNet: Removed LNI 192.168.204.11@tcp1 [ 1700.821849] Key type .llcrypt unregistered [ 1700.822992] Key type ._llcrypt unregistered [ 1702.796682] Key type ._llcrypt registered [ 1702.797904] Key type .llcrypt registered [ 1702.864691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1702.870917] alg: No test for adler32 (adler32-zlib) [ 1703.676270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1703.687441] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1706.962920] Lustre: DEBUG MARKER: Initial discovery [ 1706.987811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.111@tcp [ 1707.756820] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1707.776593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1707.793319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.111@tcp [ 1707.805382] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1707.816582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1707.831135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.111@tcp -r 1 -m GET -l 3 [ 1707.844564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.111@tcp -r 1 -m GET -e local_error [ 1707.860034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.111@tcp -r 1 -m PUT -l 6 [ 1708.312337] Lustre: DEBUG MARKER: Force oleg411-client.virtnet to discover 192.168.204.111@tcp (in background) [ 1708.333638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.111@tcp [ 1708.809483] Lustre: DEBUG MARKER: Force oleg411-server to discover 192.168.204.11@tcp [ 1708.832278] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1709.545628] Lustre: DEBUG MARKER: Wait for 70500 [ 1711.391397] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1711.849323] Lustre: DEBUG MARKER: Finished wait on 70500 [ 1711.868052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1711.884473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1711.989037] LNet: 70957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1711.991195] LNetError: 70957:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.004346] LNet: Removed LNI 192.168.204.11@tcp [ 1712.129416] Key type .llcrypt unregistered [ 1712.130756] Key type ._llcrypt unregistered [ 1717.041836] Key type ._llcrypt registered [ 1717.042756] Key type .llcrypt registered [ 1717.476132] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 04:43:43 (1769075023) [ 1717.624203] Key type .llcrypt unregistered [ 1717.625162] Key type ._llcrypt unregistered [ 1719.496273] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1719.553141] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1719.558282] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1719.560841] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1721.741615] Key type ._llcrypt registered [ 1721.742663] Key type .llcrypt registered [ 1721.800054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1721.806069] alg: No test for adler32 (adler32-zlib) [ 1722.612665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.633777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1722.642865] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1722.872235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1722.879587] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1725.642595] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 04:43:51 (1769075031) [ 1725.674384] LNet: 73018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.676540] LNetError: 73018:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.678387] LNet: Removed LNI 192.168.204.11@tcp [ 1725.685485] LNet: Removed LNI 10.1.2.3@tcp [ 1725.883597] Key type .llcrypt unregistered [ 1725.885168] Key type ._llcrypt unregistered [ 1727.899712] Key type ._llcrypt registered [ 1727.900817] Key type .llcrypt registered [ 1727.970989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1727.977242] alg: No test for adler32 (adler32-zlib) [ 1728.780806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.804463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1728.814505] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1729.082143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1729.090897] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1729.427721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.11@tcp [ 1730.464650] LNetError: Unexpected error -22 connecting to 192.168.204.11@tcp at host 192.168.204.11:988 [ 1731.540605] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 04:43:57 (1769075037) [ 1731.573954] LNet: 74144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.576812] LNetError: 74144:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.579309] LNet: Removed LNI 192.168.204.11@tcp [ 1731.587588] LNet: Removed LNI 10.1.2.3@tcp [ 1731.809422] Key type .llcrypt unregistered [ 1731.810597] Key type ._llcrypt unregistered [ 1733.736023] Key type ._llcrypt registered [ 1733.737765] Key type .llcrypt registered [ 1733.804353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1733.811640] alg: No test for adler32 (adler32-zlib) [ 1734.620237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.646665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1734.657931] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1734.919958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1734.927679] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1735.213788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.11@tcp1 --nid 192.168.204.11@tcp2 [ 1736.524963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp1 192.168.204.11@tcp2 [ 1736.534263] LNetError: 74981:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.11@tcp1 to 192.168.204.11@tcp2 - Route table may be misconfigured [ 1736.543788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp2 192.168.204.11@tcp1 [ 1738.589459] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 04:44:04 (1769075044) [ 1738.618471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.627843] LNet: 75548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.630877] LNetError: 75548:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1738.633620] LNet: Removed LNI 192.168.204.11@tcp1 [ 1739.685485] LNet: Removed LNI 192.168.204.11@tcp2 [ 1739.693571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.712302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1739.723967] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1739.981648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1739.989082] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1740.335148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1740.350215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.11@tcp [ 1740.365514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp1 192.168.204.11@tcp1 [ 1740.380052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.11@tcp -d 192.168.204.11@tcp -e network_timeout [ 1740.398842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -e network_timeout [ 1740.411920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -e network_timeout [ 1740.423238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -e network_timeout [ 1740.434436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1740.443103] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1740.450078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1740.479941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1740.513607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1740.703095] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1742.593528] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 04:44:08 (1769075048) [ 1742.631714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.643077] LNet: 76408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.646056] LNetError: 76408:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.648338] LNetError: 76408:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1742.650647] LNet: Removed LNI 192.168.204.11@tcp [ 1743.717585] LNet: Removed LNI 192.168.204.11@tcp1 [ 1743.727572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.753599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1743.833420] LNet: 76450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.959289] Key type .llcrypt unregistered [ 1743.960382] Key type ._llcrypt unregistered [ 1747.495475] Key type ._llcrypt registered [ 1747.496546] Key type .llcrypt registered [ 1747.920447] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 04:44:13 (1769075053) [ 1748.032615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1748.036992] alg: No test for adler32 (adler32-zlib) [ 1748.835503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.298430] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1749.831843] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 04:44:15 (1769075055) [ 1749.865332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.875988] LNet: 77540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.886199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.904437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1749.911774] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1750.157111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1750.163939] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1750.445709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1750.458131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp1 [ 1750.469744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp1 [ 1752.435730] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 04:44:18 (1769075058) [ 1753.000772] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg411-server [ 1753.526072] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 04:44:19 (1769075059) [ 1754.097798] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg411-server [ 1754.619341] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 04:44:20 (1769075060) [ 1755.186887] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg411-server [ 1755.755420] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 04:44:21 (1769075061) [ 1756.407561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg411-server [ 1756.981143] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 04:44:22 (1769075062) [ 1757.563440] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg411-server [ 1758.139845] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 04:44:23 (1769075063) [ 1758.777897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg411-server [ 1759.377397] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 04:44:25 (1769075065) [ 1759.995698] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg411-server [ 1760.508210] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 04:44:26 (1769075066) [ 1761.069794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg411-server [ 1761.571091] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 04:44:27 (1769075067) [ 1762.128759] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg411-server [ 1762.639391] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 04:44:28 (1769075068) [ 1762.682296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.694378] LNet: 81415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.696864] LNetError: 81415:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.698719] LNet: Removed LNI 192.168.204.11@tcp [ 1763.749390] LNet: Removed LNI 192.168.204.11@tcp1 [ 1763.756756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.777813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1763.788830] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1764.040856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1764.069350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1764.128637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.136338] LNet: 81563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.138454] LNetError: 81563:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.140517] LNetError: 81563:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1764.147268] LNet: Removed LNI 192.168.204.11@tcp [ 1764.154667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.171950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1764.180908] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1764.421079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1764.474465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.541310] LNet: Removed LNI 192.168.204.11@tcp [ 1765.548902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.566203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1765.576874] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1765.829555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1765.857606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1765.910084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.918642] LNetError: 81855:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.921202] LNetError: 81855:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1765.927415] LNet: Removed LNI 192.168.204.11@tcp [ 1765.933895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.950948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1765.962937] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1766.205741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1766.266546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.273653] LNet: 82000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1766.275524] LNet: 82000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1767.333348] LNet: Removed LNI 192.168.204.11@tcp [ 1767.341392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.357716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1767.364953] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1767.613609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1767.642691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1767.710825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.741469] LNet: Removed LNI 192.168.204.11@tcp [ 1768.750361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.772845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1768.782068] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1769.048431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1769.118155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.128196] LNetError: 82293:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.131855] LNetError: 82293:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1769.145322] LNet: Removed LNI 192.168.204.11@tcp [ 1769.154522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.173028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1769.180812] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1769.414391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1769.439926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1769.508815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.534406] LNet: Removed LNI 192.168.204.11@tcp [ 1770.543638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.566264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1770.575739] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1770.852951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1771.975767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.985231] LNet: 82594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.987884] LNet: 82594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1773.029313] LNet: Removed LNI 192.168.204.11@tcp [ 1773.038796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.060620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1773.072603] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1773.324672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1773.350459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1773.420196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.428163] LNetError: 82741:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1773.430986] LNetError: 82741:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1773.439418] LNet: Removed LNI 192.168.204.11@tcp [ 1773.445362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.464550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1773.472667] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1773.714139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1773.790606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.821283] LNet: Removed LNI 192.168.204.11@tcp [ 1774.828887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.848666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1774.856662] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1775.086200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1775.111419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1776.235538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1777.253551] LNet: Removed LNI 192.168.204.11@tcp [ 1777.265083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.289903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1777.300700] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1777.578714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1777.649024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.725297] LNet: Removed LNI 192.168.204.11@tcp [ 1778.733941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.752846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1778.762038] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1779.030512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1779.052094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.065459] LNet: Removed LNI 192.168.204.11@tcp [ 1779.073241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.089682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1779.097202] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1779.354150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1781.484397] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 04:44:47 (1769075087) [ 1781.518982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.529394] LNet: 84017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.532379] LNet: 84017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1781.535807] LNetError: 84017:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.538648] LNetError: 84017:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1781.545462] LNet: Removed LNI 192.168.204.11@tcp [ 1781.555129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.577563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1781.587200] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/180] [ 1781.960668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.981316] LNet: Removed LNI 192.168.204.11@tcp231 [ 1781.989447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.013744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1782.024189] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/180] [ 1782.027674] LNet: 84169:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1782.153026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1782.171356] LNet: Removed LNI 192.168.204.11@tcp231 [ 1782.176588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1782.184510] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/180] [ 1782.541464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.557441] LNet: Removed LNI 192.168.204.11@tcp231 [ 1782.564411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.586836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1782.595372] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/0] [ 1782.597122] LNet: 84321:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1782.721294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1782.739318] LNet: Removed LNI 192.168.204.11@tcp231 [ 1782.745440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1782.754877] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/0] [ 1783.157894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.172380] LNet: Removed LNI 192.168.204.11@tcp231 [ 1783.179541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.206086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1783.215832] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/60] [ 1783.350235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1783.365559] LNet: Removed LNI 192.168.204.11@tcp231 [ 1783.372456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1783.380897] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/60] [ 1783.766701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.783448] LNet: Removed LNI 192.168.204.11@tcp231 [ 1783.790457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.818700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1783.828548] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/180] [ 1783.830852] LNet: 84624:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1783.833148] LNet: 84624:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1783.963320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1783.981433] LNet: Removed LNI 192.168.204.11@tcp231 [ 1783.988157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1783.997260] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/180] [ 1786.444840] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 04:44:52 (1769075092) [ 1786.470896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.487451] LNet: Removed LNI 192.168.204.11@tcp231 [ 1786.497153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.518835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1786.528982] LNet: Added LNI 192.168.204.11@tcp232 [8/256/0/180] [ 1786.761540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1786.774972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1786.789329] LNet: Removed LNI 192.168.204.11@tcp232 [ 1786.795246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1786.802870] LNet: Added LNI 192.168.204.11@tcp232 [8/256/0/180] [ 1787.040975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1787.055757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1787.075381] LNet: Removed LNI 192.168.204.11@tcp232 [ 1787.082100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1787.089830] LNet: Added LNI 192.168.204.11@tcp232 [8/256/0/180] [ 1787.318555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1787.332833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1787.350441] LNet: Removed LNI 192.168.204.11@tcp232 [ 1787.356747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1787.365950] LNet: Added LNI 192.168.204.11@tcp232 [8/256/0/180] [ 1787.623399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1787.638248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1787.651648] LNet: Removed LNI 192.168.204.11@tcp232 [ 1787.657779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1789.624671] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 04:44:55 (1769075095) [ 1789.972959] Key type .llcrypt unregistered [ 1789.974049] Key type ._llcrypt unregistered [ 1791.799124] Key type ._llcrypt registered [ 1791.800271] Key type .llcrypt registered [ 1791.859290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1791.864731] alg: No test for adler32 (adler32-zlib) [ 1792.663164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1792.673663] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1795.386448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1797.870635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.882193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1797.889136] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1798.386259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_interrupt -r 2 [ 1798.397912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_interrupt -r 2 [ 1798.411381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_interrupt -r 2 [ 1798.426730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_interrupt -r 2 [ 1798.439108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_interrupt -r 2 [ 1798.450755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_interrupt -r 2 [ 1798.461769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_interrupt -r 2 [ 1798.473238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_interrupt -r 2 [ 1798.486702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1798.494599] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1798.496529] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1798.578834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1798.591981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1798.606101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1798.616944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1798.627196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1798.641019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1798.655190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1798.669562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1798.678103] LNet: 87116:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1798.686128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1798.696847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1798.708860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1798.719941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1798.732514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1798.743211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1798.917535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_dropped -r 2 [ 1798.930097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_dropped -r 2 [ 1798.942616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_dropped -r 2 [ 1798.954311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_dropped -r 2 [ 1798.965635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_dropped -r 2 [ 1798.976675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_dropped -r 2 [ 1798.989777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_dropped -r 2 [ 1799.002508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_dropped -r 2 [ 1799.014617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1799.023239] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1799.101450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1799.110996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1799.120610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1799.131035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1799.142518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1799.153985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1799.167480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1799.180653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1799.188056] LNet: 87236:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1799.194774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1799.206955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1799.219232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1799.231935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1799.242834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1799.253952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1799.437257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_aborted -r 2 [ 1799.451843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_aborted -r 2 [ 1799.463478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_aborted -r 2 [ 1799.474379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_aborted -r 2 [ 1799.485756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_aborted -r 2 [ 1799.497755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_aborted -r 2 [ 1799.510494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_aborted -r 2 [ 1799.522767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_aborted -r 2 [ 1799.533865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1799.620603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1799.633235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1799.643783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1799.653426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1799.662494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1799.671850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1799.681606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1799.693091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1799.704282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1799.717499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1799.729097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1799.741649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1799.754577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1799.765503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1799.931580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_no_route -r 2 [ 1799.945747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_no_route -r 2 [ 1799.959721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_no_route -r 2 [ 1799.971410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_no_route -r 2 [ 1799.983758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_no_route -r 2 [ 1799.996231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_no_route -r 2 [ 1800.009063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_no_route -r 2 [ 1800.024605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_no_route -r 2 [ 1800.037423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1800.047696] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1800.049815] LNet: Skipped 2 previous similar messages [ 1800.126164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1800.136123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1800.145995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1800.155524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1800.165585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1800.175745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1800.187766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1800.200087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1800.207181] LNet: 87476:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1800.210325] LNet: 87476:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1800.217408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1800.227944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1800.238688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1800.249931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1800.260623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1800.272316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1800.443308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_timeout -r 2 [ 1800.456395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_timeout -r 2 [ 1800.470328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_timeout -r 2 [ 1800.481932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_timeout -r 2 [ 1800.493509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_timeout -r 2 [ 1800.504841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_timeout -r 2 [ 1800.516365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_timeout -r 2 [ 1800.527297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_timeout -r 2 [ 1800.541561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1800.623947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1800.634171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1800.644534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1800.657796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1800.670238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1800.679610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1800.689074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1800.699211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1800.709800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1800.720700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1800.731696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1800.742723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1800.754905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1800.766200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1800.925089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_dropped -r 2 [ 1800.939230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e remote_dropped -r 2 [ 1800.950593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_dropped -r 2 [ 1800.964403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e remote_dropped -r 2 [ 1800.976346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e remote_dropped -r 2 [ 1800.990852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e remote_dropped -r 2 [ 1801.002379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e remote_dropped -r 2 [ 1801.014088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e remote_dropped -r 2 [ 1801.028060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1801.037081] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.111@tcp [ 1801.135756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1801.145377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1801.155680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1801.165089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1801.174967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1801.184305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1801.197756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1801.209981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1801.219434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1801.229264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1801.239302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1801.249785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1801.263392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1801.275953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1803.390930] LNet: 87788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.393605] LNetError: 87788:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.396496] LNet: Removed LNI 192.168.204.11@tcp [ 1803.405396] LNet: Removed LNI 192.168.204.11@tcp1 [ 1803.545665] Key type .llcrypt unregistered [ 1803.546884] Key type ._llcrypt unregistered [ 1808.371113] Key type ._llcrypt registered [ 1808.372154] Key type .llcrypt registered [ 1808.797706] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 04:45:14 (1769075114) [ 1808.920759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1808.927334] alg: No test for adler32 (adler32-zlib) [ 1809.727746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.750744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1809.758210] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1810.039320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1810.051936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.11@tcp [ 1810.058333] LNetError: 88761:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.11@tcp to recovery [ 1810.087312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.11@tcp [ 1810.125770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1812.044794] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 04:45:17 (1769075117) [ 1812.098857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1812.107595] LNet: 89349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.109473] LNetError: 89349:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1812.117597] LNet: Removed LNI 192.168.204.11@tcp [ 1812.124586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.141787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1812.149302] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1812.376225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1812.384061] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1812.696756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.11@tcp [ 1812.703460] LNetError: 89615:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.11@tcp to recovery [ 1813.151468] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1813.152234] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1813.776407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.11@tcp [ 1817.773391] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 04:45:23 (1769075123) [ 1818.320645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg411-server [ 1818.832659] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 04:45:24 (1769075124) [ 1818.864573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1818.873916] LNet: 90564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.877386] LNetError: 90564:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.880362] LNet: Removed LNI 192.168.204.11@tcp [ 1818.885643] LNet: Removed LNI 192.168.204.11@tcp1 [ 1818.895528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.916123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1818.925825] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1819.202262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1819.227728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1819.246756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1819.274160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1819.294680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1819.312341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1821.390178] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 04:45:27 (1769075127) [ 1821.426818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.437656] LNet: 91289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.440192] LNetError: 91289:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.442491] LNetError: 91289:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1821.451683] LNet: Removed LNI 192.168.204.11@tcp [ 1821.462967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.479287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.486640] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1823.728383] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 04:45:29 (1769075129) [ 1823.757856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1823.767213] LNet: 91991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.769557] LNetError: 91991:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.779658] LNet: Removed LNI 192.168.204.11@tcp [ 1823.789554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.809610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1823.817786] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1826.120371] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 04:45:31 (1769075131) [ 1826.597665] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1827.173254] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 04:45:32 (1769075132) [ 1827.366993] LNet: Removed LNI 192.168.204.11@tcp [ 1827.594614] Key type .llcrypt unregistered [ 1827.596208] Key type ._llcrypt unregistered [ 1829.579394] Key type ._llcrypt registered [ 1829.580487] Key type .llcrypt registered [ 1829.649634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1829.655409] alg: No test for adler32 (adler32-zlib) [ 1830.473584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1830.486050] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1833.560674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1834.995957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.111@tcp [ 1835.098908] LNet: 93440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.102755] LNetError: 93440:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.113314] LNet: Removed LNI 192.168.204.11@tcp [ 1835.243398] Key type .llcrypt unregistered [ 1835.244425] Key type ._llcrypt unregistered [ 1838.942393] Key type ._llcrypt registered [ 1838.943296] Key type .llcrypt registered [ 1839.528440] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1840.235812] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1840.999816] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 04:45:46 (1769075146) [ 1841.159761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1841.165516] alg: No test for adler32 (adler32-zlib) [ 1841.961542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.987205] LNet: 94524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.238396] Key type .llcrypt unregistered [ 1842.239615] Key type ._llcrypt unregistered [ 1844.475238] Key type ._llcrypt registered [ 1844.477416] Key type .llcrypt registered [ 1844.554123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1844.560449] alg: No test for adler32 (adler32-zlib) [ 1845.375352] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1845.378561] LNet: 94776:0:(config.c:1226:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1845.385686] LNet: 94776:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1845.396487] LNetError: 94788:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 1845.406799] LNetError: 94788:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.12@tcp has gone from up to down [ 1845.566864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1845.574217] LNet: 94834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.577355] LNetError: 94834:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1845.588489] LNet: Removed LNI 192.168.204.11@tcp [ 1845.597871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1845.622347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1845.634437] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1845.637505] LNetError: 94841:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.204.12@tcp is a gateway. Can not delete it [ 1845.641535] LNetError: 94841:0:(peer.c:2217:lnet_del_peer_ni()) 192.168.204.12@tcp is a router. Can not be deleted [ 1845.651353] LNet: 94841:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1847.865109] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 04:45:53 (1769075153) [ 1848.436801] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg411-server [ 1849.010290] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 04:45:54 (1769075154) [ 1849.647836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg411-server [ 1850.218392] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 04:45:55 (1769075155) [ 1850.790117] Lustre: Lustre: Build Version: 2.17.50_3_g69d656b [ 1852.447182] Key type lgssc registered [ 1853.057302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1861.977769] Key type lgssc unregistered [ 1862.191244] LNet: 97254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1862.195954] LNetError: 97254:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1862.212154] LNet: Removed LNI 192.168.204.11@tcp [ 1862.516319] Key type .llcrypt unregistered [ 1862.518106] Key type ._llcrypt unregistered [ 1867.574777] Key type ._llcrypt registered [ 1867.576122] Key type .llcrypt registered [ 1868.154120] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 04:46:13 (1769075173) [ 1868.265751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1868.272544] alg: No test for adler32 (adler32-zlib) [ 1869.077047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1869.098554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1869.112699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1869.128602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1869.143671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1871.485861] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 04:46:17 (1769075177) [ 1871.525680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1871.535265] LNet: 98652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1871.548705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.566291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1871.573895] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1871.864608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1871.871891] LNetError: 98799:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.890298] LNet: Removed LNI 192.168.204.11@tcp [ 1871.896934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.914302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1871.924915] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1872.063577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.070842] LNet: 98827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.072574] LNet: 98827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1872.078301] LNet: Removed LNI 192.168.204.11@tcp [ 1872.084656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.104259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1872.113315] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1874.648740] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 04:46:20 (1769075180) [ 1874.721576] LNet: 99412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1874.724883] LNetError: 99412:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.727835] LNetError: 99412:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1874.741088] LNet: Removed LNI 192.168.204.11@tcp [ 1874.937644] Key type .llcrypt unregistered [ 1874.939055] Key type ._llcrypt unregistered [ 1877.338318] Key type ._llcrypt registered [ 1877.340615] Key type .llcrypt registered [ 1877.444560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1877.449892] alg: No test for adler32 (adler32-zlib) [ 1878.268551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.293280] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1878.294455] LNet: *** cfs_fail_loc=e003, val=100*** [ 1878.619648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1878.630227] Ping reply received with mapping: [ 1878.631393] found metadata reply NI[192.168.204.11@tcp] [ 1878.632768] 0b cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1878.634046] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1878.635330] 5a 5a 5a 5a 5a 5a 5a 5a [ 1878.685316] LNet: 99810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.689622] LNetError: 99810:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1878.698560] LNet: Removed LNI 192.168.204.11@tcp [ 1878.961376] Key type .llcrypt unregistered [ 1878.962811] Key type ._llcrypt unregistered [ 1882.922686] Key type ._llcrypt registered [ 1882.923775] Key type .llcrypt registered [ 1883.559873] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 04:46:29 (1769075189) [ 1883.777349] Key type .llcrypt unregistered [ 1883.778471] Key type ._llcrypt unregistered [ 1885.687379] Key type ._llcrypt registered [ 1885.688511] Key type .llcrypt registered [ 1885.750109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1885.755025] alg: No test for adler32 (adler32-zlib) [ 1886.608605] LNetError: 100843:0:(api-ni.c:2732:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1886.745628] Key type .llcrypt unregistered [ 1886.746853] Key type ._llcrypt unregistered [ 1888.476169] Key type ._llcrypt registered [ 1888.477184] Key type .llcrypt registered [ 1889.077630] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 04:46:34 (1769075194) [ 1889.294266] Key type .llcrypt unregistered [ 1889.295718] Key type ._llcrypt unregistered [ 1891.240910] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1891.242317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1893.485258] Key type ._llcrypt registered [ 1893.486403] Key type .llcrypt registered [ 1893.555911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1893.560916] alg: No test for adler32 (adler32-zlib) [ 1894.369197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.392493] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1898.230762] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 04:46:43 (1769075203) [ 1898.267080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1898.277701] LNet: 102610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.280193] LNetError: 102610:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1898.292759] LNet: Removed LNI 10.1.2.3@tcp [ 1898.303390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.560293] Key type .llcrypt unregistered [ 1898.561613] Key type ._llcrypt unregistered [ 1900.737749] Key type ._llcrypt registered [ 1900.739100] Key type .llcrypt registered [ 1900.810540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1900.816262] alg: No test for adler32 (adler32-zlib) [ 1901.623952] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1901.648030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1901.699833] LNet: 102897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.702261] LNetError: 102897:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.710340] LNet: Removed LNI 192.168.204.11@tcp [ 1901.914385] Key type .llcrypt unregistered [ 1901.915954] Key type ._llcrypt unregistered [ 1905.544490] Key type ._llcrypt registered [ 1905.545652] Key type .llcrypt registered [ 1906.100221] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 04:46:51 (1769075211) [ 1906.317307] Key type .llcrypt unregistered [ 1906.319297] Key type ._llcrypt unregistered [ 1908.165641] Key type ._llcrypt registered [ 1908.166915] Key type .llcrypt registered [ 1908.229873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1908.235031] alg: No test for adler32 (adler32-zlib) [ 1911.557642] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 04:46:57 (1769075217) [ 1911.590959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1911.605822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.630791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1911.639308] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1911.909767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1911.923490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1911.938612] LNet: Removed LNI 192.168.204.11@tcp [ 1912.035045] LNet: 104630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1912.189532] Key type .llcrypt unregistered [ 1912.191222] Key type ._llcrypt unregistered [ 1915.762308] Key type ._llcrypt registered [ 1915.763216] Key type .llcrypt registered [ 1916.285372] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 04:47:01 (1769075221) [ 1916.430152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1916.436182] alg: No test for adler32 (adler32-zlib) [ 1917.256696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.279757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1917.289102] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1917.537816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1917.551249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.11@tcp [ 1917.559550] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1917.561600] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1917.563220] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1917, 0, 0 [ 1917.565429] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1917.566859] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1917.568311] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1917.569636] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1917, 0, 0 [ 1917.571473] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1917.572615] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1917.573957] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1917.575310] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1917, 0, 1967 [ 1917.577106] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1917.578262] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1917.579806] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1917.581088] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1917, 0, 1967 [ 1917.583107] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1919.932438] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 04:47:05 (1769075225) [ 1920.085746] LNet: 106164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.088386] LNetError: 106164:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.095393] LNet: Removed LNI 192.168.204.11@tcp [ 1920.312639] Key type .llcrypt unregistered [ 1920.313514] Key type ._llcrypt unregistered [ 1922.221676] Key type ._llcrypt registered [ 1922.223202] Key type .llcrypt registered [ 1922.281636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1922.286377] alg: No test for adler32 (adler32-zlib) [ 1923.087408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1923.096369] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1926.466380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1929.301359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.316347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1929.326492] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1929.791632] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1929.881318] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1929.885920] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1929.887474] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1932.083807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1932.098703] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1935.546986] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 04:47:21 (1769075241) [ 1935.602711] LNet: 107663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1935.604709] LNetError: 107663:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1935.606723] LNet: Removed LNI 192.168.204.11@tcp [ 1936.671120] LNet: Removed LNI 192.168.204.11@tcp1 [ 1936.672253] LNetError: 107663:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1936.674397] LNetError: 107663:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1936.682557] LNet: Removed LNI 10.1.2.3@tcp99 [ 1936.882451] Key type .llcrypt unregistered [ 1936.884128] Key type ._llcrypt unregistered [ 1938.809428] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1938.811272] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1940.967418] Key type ._llcrypt registered [ 1940.968460] Key type .llcrypt registered [ 1941.025960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1941.030242] alg: No test for adler32 (adler32-zlib) [ 1941.832658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.850399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1941.858225] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1942.100389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1942.108782] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1942.838866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1942.855249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1942.869408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1942.875829] LNetError: 108402:0:(peer.c:2225:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1942.886704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1942.902162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1943.022646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1943.048935] LNet: 108422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.051264] LNetError: 108422:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.053380] LNet: Removed LNI 192.168.204.11@tcp [ 1943.059310] LNet: Removed LNI 10.1.2.3@tcp [ 1943.253522] Key type .llcrypt unregistered [ 1943.254562] Key type ._llcrypt unregistered [ 1946.985406] Key type ._llcrypt registered [ 1946.986975] Key type .llcrypt registered [ 1947.406862] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 04:47:33 (1769075253) [ 1947.514408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1947.519047] alg: No test for adler32 (adler32-zlib) [ 1948.329807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1948.351324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1948.361848] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1948.605035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1948.629542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg411-client@tcp [ 1950.594667] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 04:47:36 (1769075256) [ 1950.625203] LNet: 109942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1950.628559] LNetError: 109942:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1950.639651] LNet: Removed LNI 192.168.204.11@tcp [ 1950.845775] Key type .llcrypt unregistered [ 1950.847520] Key type ._llcrypt unregistered [ 1952.624347] Key type ._llcrypt registered [ 1952.625289] Key type .llcrypt registered [ 1952.677346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1952.682389] alg: No test for adler32 (adler32-zlib) [ 1953.477295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1953.485373] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1953.495367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1955.550840] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 04:47:41 (1769075261) [ 1956.123053] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1956.658980] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 04:47:42 (1769075262) [ 1957.190256] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1957.811437] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 04:47:43 (1769075263) [ 1957.848537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1957.860090] LNet: 111355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1957.864464] LNetError: 111355:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1957.873757] LNet: Removed LNI 192.168.204.11@tcp [ 1957.883713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1957.911424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1957.923391] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1958.186322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1958.201307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1960.301107] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 04:47:46 (1769075266) [ 1960.333257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1960.342775] LNet: 112065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.345347] LNetError: 112065:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.354707] LNet: Removed LNI 192.168.204.11@tcp [ 1960.366723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1960.390801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1960.402038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1960.652288] Key type .llcrypt unregistered [ 1960.653656] Key type ._llcrypt unregistered [ 1964.433863] Key type ._llcrypt registered [ 1964.435015] Key type .llcrypt registered [ 1965.001894] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 04:47:50 (1769075270) [ 1965.113793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1965.118765] alg: No test for adler32 (adler32-zlib) [ 1965.914836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.932557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1965.940340] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1966.180256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1966.198080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1968.257677] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 04:47:53 (1769075273) [ 1968.294107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1968.302195] LNet: 113614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.304822] LNetError: 113614:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1968.313693] LNet: Removed LNI 192.168.204.11@tcp [ 1968.322951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1968.341253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1968.352944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1970.777530] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 04:47:56 (1769075276) [ 1970.815431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1970.826801] LNet: 114193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1970.839848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.883477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1971.012889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1971.026024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1971.048389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1971.165364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1971.179457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1971.209874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1971.231289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1971.263931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1971.393260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1971.406908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1971.430110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1971.550056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1971.560142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1973.866793] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 04:47:59 (1769075279) [ 1973.930523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1973.937884] LNet: 114832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.940565] LNet: 114832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1973.948518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.968365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1973.976739] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1974.203978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1974.210798] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1974.470845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1974.489944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1974.562422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1974.569198] LNetError: 115111:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1974.571109] LNet: Removed LNI 192.168.204.11@tcp [ 1974.577261] LNet: Removed LNI 192.168.204.11@tcp2 [ 1974.584369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1974.602314] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1974.604491] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1974.606468] LNet: 115117:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1974.740924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1974.810563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1975.839096] LNet: Removed LNI 192.168.204.11@tcp [ 1975.841419] LNetError: 115154:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.845114] LNetError: 115154:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1975.855443] LNet: Removed LNI 192.168.204.11@tcp2 [ 1975.866761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.894209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1975.906792] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1976.186863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1976.193821] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1976.458285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1976.473289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1976.492720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1976.560851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1976.568237] LNet: 115438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.570802] LNet: 115438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1977.631134] LNet: Removed LNI 192.168.204.11@tcp [ 1978.661501] LNet: Removed LNI 192.168.204.11@tcp2 [ 1978.672347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1978.693169] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1978.696844] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1978.699425] LNet: 115445:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1978.849275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1978.923157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1978.931597] LNetError: 115482:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1978.934022] LNetError: 115482:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1978.936675] LNet: Removed LNI 192.168.204.11@tcp [ 1980.005663] LNet: Removed LNI 192.168.204.11@tcp2 [ 1980.016336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1980.053299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1980.070782] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1980.376335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1980.387871] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1980.718182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1980.749517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1980.857153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1980.865932] LNet: 115761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.868623] LNet: 115761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1981.919191] LNet: Removed LNI 192.168.204.11@tcp [ 1981.931562] LNet: Removed LNI 192.168.204.11@tcp2 [ 1981.942671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.973203] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1981.976824] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1981.979725] LNet: 115767:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1982.147211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1982.231716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1983.263196] LNet: Removed LNI 192.168.204.11@tcp [ 1983.264768] LNetError: 115804:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1983.267072] LNetError: 115804:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1983.275498] LNet: Removed LNI 192.168.204.11@tcp2 [ 1983.284775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1983.305106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1983.316287] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1983.557417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1983.567259] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1983.907517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1983.921976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1983.944643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1984.029329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1985.055262] LNet: Removed LNI 192.168.204.11@tcp [ 1986.085383] LNet: Removed LNI 192.168.204.11@tcp2 [ 1986.095219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1986.121137] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1986.124545] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1986.127379] LNet: 116095:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1986.290921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1986.399795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1986.413486] LNet: Removed LNI 192.168.204.11@tcp [ 1987.429685] LNet: Removed LNI 192.168.204.11@tcp2 [ 1987.445425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.481881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1987.497107] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1987.880919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1987.891311] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1988.308924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1988.324163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1988.352169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1988.503204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1989.535255] LNet: Removed LNI 192.168.204.11@tcp [ 1989.543761] LNet: Removed LNI 192.168.204.11@tcp2 [ 1989.559840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.609854] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1989.615161] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1989.778690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1989.881619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1989.888366] LNet: 116486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.891092] LNet: 116486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1990.943229] LNet: Removed LNI 192.168.204.11@tcp [ 1990.952586] LNet: Removed LNI 192.168.204.11@tcp2 [ 1990.968679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1991.007253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1991.024117] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1991.433832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1991.446869] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1991.757854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1991.769721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1991.779621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1991.796891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1991.905066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1991.911665] LNetError: 116788:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1991.914073] LNetError: 116788:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1991.916731] LNet: Removed LNI 192.168.204.11@tcp [ 1992.933575] LNet: Removed LNI 192.168.204.11@tcp2 [ 1992.948753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.989866] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1992.994479] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1992.998094] LNet: 116794:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1993.002466] LNet: 116794:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1993.176289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1995.555593] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1996.170927] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 04:48:21 (1769075301) [ 1996.220600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1996.236168] LNet: Removed LNI 192.168.204.11@tcp [ 1997.289260] LNet: Removed LNI 192.168.204.11@tcp2 [ 1997.301487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1997.422109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1997.430931] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1997.665392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1997.675254] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1997.946989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1997.954291] LNet: Removed LNI 192.168.204.11@tcp [ 1997.959346] LNet: Removed LNI 10.1.2.3@tcp [ 1997.965974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1997.987379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1997.996656] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1997.998391] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1998.128805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1998.137380] LNet: Removed LNI 192.168.204.11@tcp [ 1998.147319] LNet: Removed LNI 10.1.2.3@tcp [ 2000.312931] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 04:48:26 (1769075306) [ 2000.347715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2000.367300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2000.403422] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2000.446592] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2000.449141] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2000.451577] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2002.617809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2002.629353] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2002.943208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2002.956865] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2003.647307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2003.663493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2003.684176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.11@tcp -m PUT -e local_timeout -r 1 [ 2003.708857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2003.731389] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2003.732976] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2003.806482] LNetError: 118491:0:(lib-move.c:4947:lnet_parse()) 192.168.204.11@tcp, src 192.168.204.11@tcp: Dropping PUT (error -108 looking up sender) [ 2005.855084] LNet: Removed LNI 192.168.204.11@tcp [ 2005.862352] LNet: Removed LNI 10.1.2.3@tcp [ 2006.035248] Key type .llcrypt unregistered [ 2006.036287] Key type ._llcrypt unregistered [ 2009.828295] Key type ._llcrypt registered [ 2009.829283] Key type .llcrypt registered [ 2010.275954] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 04:48:35 (1769075315) [ 2010.393475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2010.398693] alg: No test for adler32 (adler32-zlib) [ 2011.218513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2011.373693] LNet: 119797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.655352] Key type .llcrypt unregistered [ 2011.656733] Key type ._llcrypt unregistered [ 2013.675271] Key type ._llcrypt registered [ 2013.676463] Key type .llcrypt registered [ 2013.744923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2013.750316] alg: No test for adler32 (adler32-zlib) [ 2014.591387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2014.609328] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2017.432056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 2017.479777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2017.653197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.111@tcp [ 2017.695063] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.111@tcp [ 2017.725814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2017.735908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2017.745425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2017.755090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2017.765287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2017.774984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2017.785702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2017.796394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2017.802566] LNet: 120241:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2017.808924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2017.819153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2017.829528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2017.840192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2017.850605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2017.860413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2018.061809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2018.136117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2018.146403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2018.156100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2018.166819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2018.177992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2018.188240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2018.198712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2018.210394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2018.222559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2018.235348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2018.247676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2018.260239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2018.273330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2018.287286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2018.491442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.111@tcp [ 2018.562420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2018.572236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2018.581616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2018.591062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2018.600980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2018.610324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2018.619749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2018.629670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2018.635554] LNet: 120475:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2018.637573] LNet: 120475:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 2018.643377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2018.654388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2018.666552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2018.676270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2018.685821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2018.696533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2018.921115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2019.006671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2019.018576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2019.028987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2019.039605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2019.049078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2019.058266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2019.068158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2019.077995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2019.088567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2019.099897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2019.109814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2019.119981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2019.130548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2019.141601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2019.240770] LNet: 120637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.243282] LNetError: 120637:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.249344] LNet: Removed LNI 192.168.204.11@tcp [ 2019.376517] Key type .llcrypt unregistered [ 2019.377474] Key type ._llcrypt unregistered [ 2024.228458] Key type ._llcrypt registered [ 2024.229738] Key type .llcrypt registered [ 2024.729290] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 04:48:50 (1769075330) [ 2024.859846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2024.864985] alg: No test for adler32 (adler32-zlib) [ 2025.667527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2025.799306] LNet: 121534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2026.016292] Key type .llcrypt unregistered [ 2026.017370] Key type ._llcrypt unregistered [ 2028.163232] Key type ._llcrypt registered [ 2028.164624] Key type .llcrypt registered [ 2028.218909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2028.223227] alg: No test for adler32 (adler32-zlib) [ 2029.024116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2029.034966] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2032.222104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 2034.710640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.723773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2034.732428] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 2035.045327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2035.259457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.111@tcp [ 2035.354971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2035.365853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2035.376288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2035.387651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2035.401362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2035.411464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2035.422184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2035.431841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2035.437942] LNet: 122180:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2035.444822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2035.454358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2035.464823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2035.474510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2035.486861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2035.498488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2035.739101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2035.835543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2035.846478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2035.857745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2035.868705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2035.879863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2035.890392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2035.901798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2035.912482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2035.923687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2035.937333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2035.948509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2035.959951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2035.971495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2035.987382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2036.218542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.111@tcp [ 2036.313235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2036.323802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2036.334714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2036.345297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2036.355772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2036.366920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2036.378090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2036.389095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2036.396684] LNet: 122414:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2036.399724] LNet: 122414:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 2036.407717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2036.422599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2036.434460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2036.447482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2036.460384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2036.474826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2036.727847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2036.817573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2036.828374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2036.838772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2036.849486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2036.860399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2036.871031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2036.883202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2036.894955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2036.906363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2036.918378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2036.928869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2036.939692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2036.951369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2036.963266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2039.080633] LNet: 122610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2039.082985] LNetError: 122610:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2039.085725] LNet: Removed LNI 192.168.204.11@tcp [ 2039.094398] LNet: Removed LNI 192.168.204.11@tcp1 [ 2039.240886] Key type .llcrypt unregistered [ 2039.241961] Key type ._llcrypt unregistered [ 2044.151804] Key type ._llcrypt registered [ 2044.153345] Key type .llcrypt registered [ 2044.556945] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 04:49:10 (1769075350) [ 2046.790694] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 04:49:12 (1769075352) [ 2048.997775] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 04:49:14 (1769075354) [ 2051.293334] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 04:49:17 (1769075357) [ 2053.727516] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 04:49:19 (1769075359) [ 2055.876263] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 04:49:21 (1769075361) [ 2058.130974] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 04:49:23 (1769075363) [ 2060.414831] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 04:49:26 (1769075366) [ 2062.776728] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 04:49:28 (1769075368) [ 2064.917241] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 04:49:30 (1769075370) [ 2067.158272] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 04:49:32 (1769075372) [ 2069.518286] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 04:49:35 (1769075375) [ 2072.019850] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 04:49:37 (1769075377) [ 2074.331759] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 04:49:40 (1769075380) [ 2076.601363] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 04:49:42 (1769075382) [ 2078.876195] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 04:49:44 (1769075384) [ 2081.053066] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 04:49:46 (1769075386) [ 2083.427772] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 04:49:49 (1769075389) [ 2083.573931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.579404] alg: No test for adler32 (adler32-zlib) [ 2084.380640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.423747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2084.433907] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2084.797239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2084.803430] LNet: 133299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2084.805870] LNetError: 133299:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2084.819256] LNet: Removed LNI 192.168.204.11@tcp [ 2084.826508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.843992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2084.851650] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2086.974819] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 04:49:52 (1769075392) [ 2089.449059] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 04:49:55 (1769075395) [ 2091.635271] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 04:49:57 (1769075397) [ 2093.829547] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 04:49:59 (1769075399) [ 2096.002932] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 04:50:01 (1769075401) [ 2098.349227] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 04:50:04 (1769075404) [ 2100.533861] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 04:50:06 (1769075406) [ 2102.943639] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 04:50:08 (1769075408) [ 2105.254396] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 04:50:10 (1769075410) [ 2107.657616] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 04:50:13 (1769075413) [ 2110.005423] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 04:50:15 (1769075415) [ 2112.211959] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 04:50:17 (1769075417) [ 2114.347702] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1808 sec ========== 04:50:20 (1769075420) [ 2114.435558] LNet: 140754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2114.437789] LNetError: 140754:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2114.444536] LNet: Removed LNI 192.168.204.11@tcp [ 2114.633492] Key type .llcrypt unregistered [ 2114.634593] Key type ._llcrypt unregistered [ 2118.867320] Key type ._llcrypt registered [ 2118.869253] Key type .llcrypt registered [ 2119.021552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2119.027838] alg: No test for adler32 (adler32-zlib) [ 2119.870555] Lustre: Lustre: Build Version: 2.17.50_3_g69d656b [ 2119.951318] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2121.535208] Key type lgssc registered [ 2122.202611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2145.944594] Lustre: Mounted lustre-client [ 2148.406436] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2155.794071] LustreError: 143208:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff893885059800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2155.799600] LustreError: 143208:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2155.812043] Lustre: Unmounted lustre-client [ 2168.226214] Key type lgssc unregistered [ 2168.358874] LNet: 143689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.362658] LNetError: 143689:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.373929] LNet: Removed LNI 192.168.204.11@tcp [ 2168.693108] Key type .llcrypt unregistered [ 2168.694506] Key type ._llcrypt unregistered