[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 578035826 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002374] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006997] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009139] LSM: Security Framework initializing [ 0.010044] Yama: becoming mindful. [ 0.011034] SELinux: Initializing. [ 0.012052] *** VALIDATE selinux *** [ 0.020608] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024478] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026097] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027105] *** VALIDATE tmpfs *** [ 0.029327] *** VALIDATE proc *** [ 0.030204] *** VALIDATE cgroup *** [ 0.031007] *** VALIDATE cgroup2 *** [ 0.032256] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033140] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035026] Spectre V2 : User space: Vulnerable [ 0.036007] Speculative Store Bypass: Vulnerable [ 0.039104] debug: unmapping init [mem 0xffffffff89a59000-0xffffffff89a60fff] [ 0.041821] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042671] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043025] ... version: 2 [ 0.044009] ... bit width: 48 [ 0.045009] ... generic registers: 4 [ 0.046009] ... value mask: 0000ffffffffffff [ 0.047011] ... max period: 00007fffffffffff [ 0.048011] ... fixed-purpose events: 3 [ 0.049010] ... event mask: 000000070000000f [ 0.050282] rcu: Hierarchical SRCU implementation. [ 0.052363] smp: Bringing up secondary CPUs ... [ 0.053556] x86: Booting SMP configuration: [ 0.054015] .... node #0, CPUs: #1 #2 #3 [ 0.057426] smp: Brought up 1 node, 4 CPUs [ 0.059011] smpboot: Max logical packages: 1 [ 0.060027] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.127021] node 0 deferred pages initialised in 63ms [ 0.130129] devtmpfs: initialized [ 0.131240] x86/mm: Memory block size: 128MB [ 0.134326] gcov: version magic: 0x41383552 [ 0.137293] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.138075] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.139243] pinctrl core: initialized pinctrl subsystem [ 0.140207] [ 0.140856] ************************************************************* [ 0.141014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.142013] ** ** [ 0.143011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.144013] ** ** [ 0.145012] ** This means that this kernel is built to expose internal ** [ 0.146012] ** IOMMU data structures, which may compromise security on ** [ 0.147015] ** your system. ** [ 0.148013] ** ** [ 0.149012] ** If you see this message and you are not debugging the ** [ 0.150014] ** kernel, report this immediately to your vendor! ** [ 0.151013] ** ** [ 0.152014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153011] ************************************************************* [ 0.154715] NET: Registered protocol family 16 [ 0.155426] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.156052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.157064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.158443] cpuidle: using governor menu [ 0.160000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.162497] PCI: Using configuration type 1 for base access [ 0.164143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.173085] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.174051] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.176109] cryptd: max_cpu_qlen set to 1000 [ 0.178391] ACPI: Added _OSI(Module Device) [ 0.180019] ACPI: Added _OSI(Processor Device) [ 0.182017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.184017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.189972] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.198425] ACPI: Interpreter enabled [ 0.199050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.200010] ACPI: Using IOAPIC for interrupt routing [ 0.202087] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.205434] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.214830] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.219041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.224070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.229351] acpiphp: Slot [2] registered [ 0.230131] acpiphp: Slot [5] registered [ 0.231095] acpiphp: Slot [6] registered [ 0.233130] acpiphp: Slot [3] registered [ 0.234086] acpiphp: Slot [4] registered [ 0.235080] acpiphp: Slot [7] registered [ 0.236074] acpiphp: Slot [8] registered [ 0.238073] acpiphp: Slot [9] registered [ 0.239080] acpiphp: Slot [10] registered [ 0.241097] acpiphp: Slot [11] registered [ 0.242800] acpiphp: Slot [12] registered [ 0.245094] acpiphp: Slot [13] registered [ 0.246074] acpiphp: Slot [14] registered [ 0.248068] acpiphp: Slot [15] registered [ 0.249126] acpiphp: Slot [16] registered [ 0.251096] acpiphp: Slot [17] registered [ 0.252060] acpiphp: Slot [18] registered [ 0.253119] acpiphp: Slot [19] registered [ 0.254076] acpiphp: Slot [20] registered [ 0.256224] acpiphp: Slot [21] registered [ 0.258133] acpiphp: Slot [22] registered [ 0.259195] acpiphp: Slot [23] registered [ 0.261116] acpiphp: Slot [24] registered [ 0.262091] acpiphp: Slot [25] registered [ 0.264079] acpiphp: Slot [26] registered [ 0.265073] acpiphp: Slot [27] registered [ 0.266081] acpiphp: Slot [28] registered [ 0.268120] acpiphp: Slot [29] registered [ 0.269080] acpiphp: Slot [30] registered [ 0.270072] acpiphp: Slot [31] registered [ 0.271053] PCI host bridge to bus 0000:00 [ 0.273012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.275014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.278026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.280018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.283021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.286046] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.287242] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.291155] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.294647] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.302013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.306390] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.310031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.312013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.314011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.316678] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.319941] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.323044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.327077] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.333011] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.344013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.349012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.355759] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.362012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.368014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.379014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.395919] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.408013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.416020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.437019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.455000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.458332] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.460308] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.465197] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.466394] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.473131] iommu: Default domain type: Passthrough [ 0.475461] SCSI subsystem initialized [ 0.477116] ACPI: bus type USB registered [ 0.478093] usbcore: registered new interface driver usbfs [ 0.480063] usbcore: registered new interface driver hub [ 0.482065] usbcore: registered new device driver usb [ 0.483158] pps_core: LinuxPPS API ver. 1 registered [ 0.485008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.488202] PTP clock support registered [ 0.490193] EDAC MC: Ver: 3.0.0 [ 0.492017] PCI: Using ACPI for IRQ routing [ 0.493000] NetLabel: Initializing [ 0.495012] NetLabel: domain hash size = 128 [ 0.497012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.499122] NetLabel: unlabeled traffic allowed by default [ 0.501181] vgaarb: loaded [ 0.503411] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.504015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.508304] clocksource: Switched to clocksource kvm-clock [ 0.624113] VFS: Disk quotas dquot_6.6.0 [ 0.628775] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.631634] *** VALIDATE ramfs *** [ 0.633071] *** VALIDATE hugetlbfs *** [ 0.634968] pnp: PnP ACPI init [ 0.639082] pnp: PnP ACPI: found 6 devices [ 0.656785] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.660978] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.663886] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.669421] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.671796] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.674107] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.676945] NET: Registered protocol family 2 [ 0.679490] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.684461] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.687711] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.693444] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.696457] TCP: Hash tables configured (established 65536 bind 65536) [ 0.699509] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.702782] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.705787] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.709431] NET: Registered protocol family 1 [ 0.712316] RPC: Registered named UNIX socket transport module. [ 0.714850] RPC: Registered udp transport module. [ 0.719285] RPC: Registered tcp transport module. [ 0.721294] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.723365] NET: Registered protocol family 44 [ 0.724877] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.727074] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.729071] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.731235] PCI: CLS 0 bytes, default 64 [ 0.732573] Unpacking initramfs... [ 2.158977] debug: unmapping init [mem 0xffff97e7bcc64000-0xffff97e7bffcffff] [ 2.166339] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.168717] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.172632] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.687982] Initialise system trusted keyrings [ 2.692929] Key type blacklist registered [ 2.695215] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.704457] zbud: loaded [ 2.707709] *** VALIDATE nfs *** [ 2.709042] *** VALIDATE nfs4 *** [ 2.711530] pstore: using deflate compression [ 2.715471] Platform Keyring initialized [ 2.849479] NET: Registered protocol family 38 [ 2.851420] Key type asymmetric registered [ 2.852949] Asymmetric key parser 'x509' registered [ 2.855115] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.859533] io scheduler mq-deadline registered [ 2.861321] io scheduler kyber registered [ 2.863170] io scheduler bfq registered [ 2.865113] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.868516] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.871565] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.875252] ACPI: Power Button [PWRF] [ 2.881202] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.887939] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.917882] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.947051] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.974647] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.979801] Non-volatile memory driver v1.3 [ 2.981842] Linux agpgart interface v0.103 [ 3.012661] virtio_blk virtio1: [vda] 145928 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.015554] vda: detected capacity change from 0 to 74715136 [ 3.047833] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.051505] vdb: detected capacity change from 0 to 1073741824 [ 3.060039] libphy: Fixed MDIO Bus: probed [ 3.065826] usbcore: registered new interface driver usbserial_generic [ 3.068527] usbserial: USB Serial support registered for generic [ 3.071245] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.075433] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.077536] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.080120] mousedev: PS/2 mouse device common for all mice [ 3.083850] rtc_cmos 00:05: RTC can wake from S4 [ 3.087615] rtc_cmos 00:05: registered as rtc0 [ 3.089606] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.093064] intel_pstate: CPU model not supported [ 3.096657] hid: raw HID events driver (C) Jiri Kosina [ 3.099421] usbcore: registered new interface driver usbhid [ 3.102747] usbhid: USB HID core driver [ 3.104767] drop_monitor: Initializing network drop monitor service [ 3.105492] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.110016] Initializing XFRM netlink socket [ 3.111462] NET: Registered protocol family 10 [ 3.118112] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.122789] Segment Routing with IPv6 [ 3.127482] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.127497] NET: Registered protocol family 17 [ 3.133338] mpls_gso: MPLS GSO support [ 3.139321] RAS: Correctable Errors collector initialized. [ 3.141732] AVX version of gcm_enc/dec engaged. [ 3.143553] AES CTR mode by8 optimization enabled [ 3.235430] sched_clock: Marking stable (3235407932, 0)->(4125289265, -889881333) [ 3.239248] registered taskstats version 1 [ 3.241369] Loading compiled-in X.509 certificates [ 3.243351] zswap: loaded using pool lzo/zbud [ 3.272074] Key type big_key registered [ 3.287080] Key type encrypted registered [ 3.288427] ima: No TPM chip found, activating TPM-bypass! [ 3.290441] ima: Allocated hash algorithm: sha1 [ 3.292033] ima: No architecture policies found [ 3.293843] evm: Initialising EVM extended attributes: [ 3.295747] evm: security.selinux [ 3.296864] evm: security.ima [ 3.297976] evm: security.capability [ 3.299329] evm: HMAC attrs: 0x1 [ 3.301765] rtc_cmos 00:05: setting system clock to 2026-07-30 05:32:50 UTC (1785389570) [ 3.308516] debug: unmapping init [mem 0xffffffff8aa03000-0xffffffff8abfffff] [ 3.312283] debug: unmapping init [mem 0xffffffff89782000-0xffffffff89a58fff] [ 3.321078] Write protecting the kernel read-only data: 28672k [ 3.324499] debug: unmapping init [mem 0xffffffff87e03000-0xffffffff87ffffff] [ 3.327179] debug: unmapping init [mem 0xffffffff88714000-0xffffffff887fffff] [ 3.355924] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.363532] systemd[1]: Detected virtualization kvm. [ 3.364923] systemd[1]: Detected architecture x86-64. [ 3.367057] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.390436] systemd[1]: No hostname configured. [ 3.392562] systemd[1]: Set hostname to . [ 3.395059] random: systemd: uninitialized urandom read (16 bytes read) [ 3.398073] systemd[1]: Initializing machine ID from random generator. [ 3.454687] random: ln: uninitialized urandom read (6 bytes read) [ 3.541975] random: systemd: uninitialized urandom read (16 bytes read) [ 3.545294] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.552678] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.561859] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.141490] device-mapper: uevent: version 1.0.3 [ 4.144230] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.880637] virtio_net virtio0 ens2: renamed from eth0 [ 4.963668] scsi host0: ata_piix [ 4.970884] scsi host1: ata_piix [ 4.972249] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.974273] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 7.056276] random: fast init done [ 8.643833] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.803652] random: crng init done [ 9.804867] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.211137] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.286630] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.571129] SELinux: Disabled at runtime. [ 11.629721] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.637775] systemd[1]: Detected virtualization kvm. [ 11.639308] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.215285] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.218736] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.223946] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.228182] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.237257] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.247881] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.257207] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Starting Create [ 12.306394] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.799571] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.191747] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.311537] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.414047] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.438958] EDAC sbridge: Ver: 1.1.2 [ 14.695568] Key type dns_resolver registered [ 15.010332] NFS: Registering the id_resolver key type [ 15.016573] Key type id_resolver registered [ 15.019520] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg429-client login: [ 110.687765] libcfs: loading out-of-tree module taints kernel. [ 111.245526] Key type ._llcrypt registered [ 111.253909] Key type .llcrypt registered [ 111.884212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 111.915250] alg: No test for adler32 (adler32-zlib) [ 113.646505] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 114.679699] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 116.433876] Key type lgssc registered [ 117.416879] hrtimer: interrupt took 32872006 ns [ 118.811649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 302.112359] Lustre: Mounted lustre-client [ 308.466712] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 325.769207] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing check_logdir /tmp/testlogs/ [ 327.655181] Lustre: lustre-OST0000-osc-ffff97e8047f3000: disconnect after 22s idle [ 332.210444] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing yml_node [ 339.660466] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 342.961255] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 347.261422] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 350.365148] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jul 30 01:38:34 EDT 2026 [ 377.024576] Lustre: Unmounted lustre-client [ 411.370678] Key type lgssc unregistered [ 411.796590] LNet: 8405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.805255] LNetError: 8405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 411.831683] LNet: Removed LNI 192.168.204.29@tcp [ 413.443279] Key type .llcrypt unregistered [ 413.448264] Key type ._llcrypt unregistered [ 450.943883] Key type ._llcrypt registered [ 450.950366] Key type .llcrypt registered [ 451.410103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 451.431902] alg: No test for adler32 (adler32-zlib) [ 452.503724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 452.649827] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 452.699952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 458.844354] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 459.204420] LNet: 9291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 459.217327] LNetError: 9291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 459.251188] LNet: Removed LNI 192.168.204.29@tcp [ 460.201774] Key type .llcrypt unregistered [ 460.208366] Key type ._llcrypt unregistered [ 470.451544] Key type ._llcrypt registered [ 470.455388] Key type .llcrypt registered [ 472.412606] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 01:40:37 (1785390037) [ 472.881268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 472.903988] alg: No test for adler32 (adler32-zlib) [ 474.086954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.250929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.272808] LNet: 9703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.302268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.381916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 474.396718] LNet: 9710:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 485.357765] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 01:40:49 (1785390049) [ 485.547462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.586354] LNet: 10290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.627196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.692324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 497.247229] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 01:41:01 (1785390061) [ 497.427735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.474563] LNet: 10875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.529050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.684308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 506.800504] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 01:41:12 (1785390072) [ 507.049191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.103930] LNet: 11457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 507.188901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.285464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 516.922703] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 01:41:21 (1785390081) [ 517.072754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.149701] LNet: 12040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.211863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.456200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 518.166495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 529.690554] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 01:41:33 (1785390093) [ 530.164426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.196226] LNet: 12637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 530.232544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.398154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 531.101454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 531.889372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 541.350652] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 01:41:46 (1785390106) [ 542.041697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.152529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.318445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 556.340639] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 01:42:01 (1785390121) [ 556.544280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.625272] LNet: 13813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 556.641702] LNet: 13813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 556.722924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.092133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 557.292511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 558.795184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 558.905273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 559.685838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 559.743690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 560.472503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 560.538673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 560.970768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 561.026199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 561.586496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 561.645452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 562.108880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 562.150785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 562.497377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 562.549897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 562.973631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 563.013256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 563.821479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 563.869336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 564.637718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 564.717213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 565.549984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 565.618653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 566.290351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 566.342533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 576.870487] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 01:42:22 (1785390142) [ 577.042671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.113822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.262052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 577.323238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 586.810447] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 01:42:32 (1785390152) [ 587.363172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.437837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.588351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 587.672554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 597.442597] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 01:42:42 (1785390162) [ 597.609241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.661074] LNet: 15677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.670165] LNet: 15677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 597.699110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.788724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 597.818476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 604.630759] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 01:42:50 (1785390170) [ 604.945666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.988808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.072707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 605.097696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 613.421497] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 01:42:58 (1785390178) [ 613.691965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.752853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.856942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 613.928150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 623.056965] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 01:43:07 (1785390187) [ 623.170542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.243532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.407361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 623.500322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 635.205415] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 01:43:19 (1785390199) [ 635.450446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.545242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.790876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 636.498919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 649.560870] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 01:43:33 (1785390213) [ 649.812986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.940250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.585763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 651.276936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 662.227801] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 01:43:47 (1785390227) [ 662.462110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.518550] LNet: 19239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.531413] LNet: 19239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 662.576438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.882153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 663.314381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 675.179880] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 01:43:59 (1785390239) [ 675.323387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.397733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.340375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 676.891336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 687.417545] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 01:44:12 (1785390252) [ 687.585133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.680646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.834910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 687.893352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 696.146194] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 01:44:22 (1785390262) [ 696.372341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.437923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.555714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.653635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 706.659857] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 01:44:31 (1785390271) [ 706.832576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.916155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.112502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 716.832161] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 01:44:41 (1785390281) [ 717.066582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.155083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.354428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 727.639683] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 01:44:52 (1785390292) [ 727.747464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.839000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.934861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 729.017726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 739.859880] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 01:45:04 (1785390304) [ 740.006314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.121274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.310876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 740.379524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 749.978265] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:45:14 (1785390314) [ 750.087710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.163426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.318361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 750.370676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 759.319315] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:45:24 (1785390324) [ 759.759847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.840039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.977336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 760.051754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 769.070981] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:45:34 (1785390334) [ 769.758965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 769.879587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.011504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 770.106584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 782.487971] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:45:46 (1785390346) [ 782.771613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 782.907214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.028953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 783.162277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 783.226843] LNetError: 25837:0:(peer.c:2277:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 783.314810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 783.411276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 794.423851] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:45:58 (1785390358) [ 794.537781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.566239] LNet: 26408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.571347] LNet: 26408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 794.626338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.830356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.EVCEKBoK1Q [ 794.919772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.EVCEKBoK1Q [ 794.974961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.EVCEKBoK1Q [ 804.001923] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:46:09 (1785390369) [ 804.373242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.495958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.606777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 804.657525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 804.712373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 811.687419] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 01:46:17 (1785390377) [ 812.055901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.114819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.184744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 812.208969] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 813.830587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 815.717824] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 817.898450] Key type lgssc registered [ 819.860477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 825.433188] LNetError: 27601:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.29@tcp99 has no available nets [ 829.993423] Key type lgssc unregistered [ 830.651642] LNetError: 28710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.718531] LNet: Removed LNI 192.168.204.29@tcp [ 831.979518] Key type .llcrypt unregistered [ 831.985264] Key type ._llcrypt unregistered [ 841.585571] Key type ._llcrypt registered [ 841.590493] Key type .llcrypt registered [ 843.463358] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:46:48 (1785390408) [ 843.913890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 843.929802] alg: No test for adler32 (adler32-zlib) [ 845.066834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.232639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 849.421624] LNet: Message forwarding will be enabled in 2s [ 852.512167] LNet: Message forwarding enabled [ 853.554539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 853.606432] LNet: Message forwarding will stop in 2s to 6s [ 858.648539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 858.675332] LNet: Message forwarding disabled [ 862.187020] LNet: Message forwarding will be enabled in 2s [ 865.830681] LNet: Message forwarding enabled [ 871.633656] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:47:17 (1785390437) [ 871.808559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.858323] LNet: 30113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.370854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.442739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 884.448493] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:47:29 (1785390449) [ 884.615680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.722305] LNet: 30697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.812770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.979831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 885.078569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 885.147324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 885.209132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 885.287857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 885.389934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 885.447409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 885.533804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 885.603421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 885.665138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 885.722105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 885.814127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 885.888846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 885.980497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 886.052065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 886.161701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 886.284197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 886.349488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 886.404428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 886.439334] LNetError: 30747:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 897.410438] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:47:42 (1785390462) [ 897.643739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.701681] LNet: 31314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.729183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.936448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 908.050272] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:47:53 (1785390473) [ 910.980423] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg429-server [ 914.134485] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:47:58 (1785390478) [ 914.325276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.393192] LNet: 32247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.438097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.510668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 914.568454] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 916.219576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[30-34]@tcp [ 916.267122] LNet: 32387:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 916.340716] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.32@tcp is being used as a gateway but routing feature is not turned on [ 916.365573] LNetError: 32257:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.32@tcp has gone from up to down [ 917.350808] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.32@tcp is being used as a gateway but routing feature is not turned on [ 917.371635] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 918.380228] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.32@tcp is being used as a gateway but routing feature is not turned on [ 918.387016] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 920.440312] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.30@tcp is being used as a gateway but routing feature is not turned on [ 920.453062] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 924.532741] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.30@tcp is being used as a gateway but routing feature is not turned on [ 924.548239] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 926.205517] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:48:11 (1785390491) [ 929.004102] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg429-server [ 931.356790] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:48:16 (1785390496) [ 931.554062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.588475] LNet: 33324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.596320] LNetError: 33324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.630472] LNet: Removed LNI 192.168.204.29@tcp [ 931.725052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.840788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 931.884212] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 932.958692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[30-35/2]@tcp [ 932.994626] LNet: 33441:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 933.008862] LNet: 33441:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 933.059346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[30-35/2]@tcp [ 933.062399] LNetError: 33328:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.34@tcp is being used as a gateway but routing feature is not turned on [ 933.112854] LNetError: 33328:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 34 previous similar messages [ 933.130057] LNetError: 33328:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.34@tcp has gone from up to down [ 933.143910] LNetError: 33328:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 942.145187] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:48:27 (1785390507) [ 942.524132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 942.566173] LNet: 34033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.573679] LNetError: 34033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.617504] LNet: Removed LNI 192.168.204.29@tcp [ 942.674813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.847410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 942.937523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 942.999932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.177506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 943.318965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.430120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.549398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 943.709443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.790285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.937153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 944.021981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.070932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.173586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 944.403397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.483333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.592135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 944.787423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.874210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.949862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 945.074434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.147510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.283081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 945.390239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.465335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.548162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 945.706775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.793510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.924599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 946.075194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.124717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.200367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 946.246189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.275070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.331569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 954.267332] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:48:39 (1785390519) [ 956.365260] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg429-server [ 958.618832] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:48:44 (1785390524) [ 961.417899] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg429-server [ 963.379155] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:48:49 (1785390529) [ 963.872548] LNet: 35435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.881919] LNet: 35435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 964.722357] Key type .llcrypt unregistered [ 964.725519] Key type ._llcrypt unregistered [ 973.951961] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 974.102228] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 974.116309] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 974.130444] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 977.085349] Key type ._llcrypt registered [ 977.100543] Key type .llcrypt registered [ 977.372281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 977.404471] alg: No test for adler32 (adler32-zlib) [ 978.372854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.446382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 978.497274] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 979.388066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 979.430143] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 982.106564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 982.162730] LNet: Removed LNI 10.1.2.3@tcp [ 982.407267] LNet: 36158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.428059] LNetError: 36158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.437155] LNet: Removed LNI 192.168.204.29@tcp [ 983.144526] Key type .llcrypt unregistered [ 983.145943] Key type ._llcrypt unregistered [ 999.334054] Key type ._llcrypt registered [ 999.336946] Key type .llcrypt registered [ 1001.496322] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:49:26 (1785390566) [ 1002.566944] Key type .llcrypt unregistered [ 1002.574096] Key type ._llcrypt unregistered [ 1012.951697] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1013.172376] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1013.179647] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1013.190434] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1016.083819] Key type ._llcrypt registered [ 1016.085869] Key type .llcrypt registered [ 1016.269605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1016.287876] alg: No test for adler32 (adler32-zlib) [ 1017.153629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1017.198933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1017.214287] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1017.673766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1017.693480] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1020.104160] LNet: 37654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.109114] LNetError: 37654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1020.114493] LNet: Removed LNI 10.1.2.3@tcp [ 1020.121732] LNet: Removed LNI 192.168.204.29@tcp [ 1020.736505] Key type .llcrypt unregistered [ 1020.744978] Key type ._llcrypt unregistered [ 1035.688842] Key type ._llcrypt registered [ 1035.692822] Key type .llcrypt registered [ 1038.194378] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:50:02 (1785390602) [ 1038.815472] Key type .llcrypt unregistered [ 1038.822393] Key type ._llcrypt unregistered [ 1048.643809] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1048.797181] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1048.814424] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1048.826899] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1051.941365] Key type ._llcrypt registered [ 1051.946962] Key type .llcrypt registered [ 1052.342395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1052.375350] alg: No test for adler32 (adler32-zlib) [ 1053.467619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1053.628850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 1053.699590] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1057.153872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1057.171223] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1059.873575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 1059.901589] LNet: Removed LNI 10.1.2.3@tcp [ 1059.931113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 1059.970768] LNet: Removed LNI 10.1.2.31@tcp [ 1059.988438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1060.039181] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1060.778414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 1060.797678] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1061.784929] LNet: 39579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.795103] LNetError: 39579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 1061.800486] LNet: Removed LNI 10.1.2.31@tcp [ 1061.809781] LNet: Removed LNI 10.1.2.3@tcp [ 1062.465648] Key type .llcrypt unregistered [ 1062.467301] Key type ._llcrypt unregistered [ 1080.453691] Key type ._llcrypt registered [ 1080.459658] Key type .llcrypt registered [ 1081.673566] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:50:47 (1785390647) [ 1082.147490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1082.161638] alg: No test for adler32 (adler32-zlib) [ 1083.148348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.197376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1083.224825] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1084.099727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1084.126735] LNet: 40534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.134323] LNetError: 40534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.147706] LNet: Removed LNI 192.168.204.29@tcp [ 1084.166154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1084.273205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.29@tcp [ 1084.293836] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1084.969430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.29@tcp [ 1084.992660] LNet: Removed LNI 192.168.204.29@tcp [ 1085.070244] LNet: 40669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.820421] Key type .llcrypt unregistered [ 1085.822838] Key type ._llcrypt unregistered [ 1101.268368] Key type ._llcrypt registered [ 1101.273359] Key type .llcrypt registered [ 1102.950713] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:51:08 (1785390668) [ 1106.723157] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg429-server [ 1109.825829] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:51:14 (1785390674) [ 1110.750405] Key type .llcrypt unregistered [ 1110.751786] Key type ._llcrypt unregistered [ 1120.326710] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1120.527136] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1120.537845] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1120.549527] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1123.243653] Key type ._llcrypt registered [ 1123.245775] Key type .llcrypt registered [ 1123.405627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1123.420163] alg: No test for adler32 (adler32-zlib) [ 1124.510198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1124.633670] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1125.996297] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 1128.658760] LNet: 42509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.673893] LNetError: 42509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1128.681752] LNet: Removed LNI 192.168.204.29@tcp1 [ 1128.704256] LNet: Removed LNI 10.1.2.3@tcp2 [ 1129.595467] Key type .llcrypt unregistered [ 1129.599972] Key type ._llcrypt unregistered [ 1147.875792] Key type ._llcrypt registered [ 1147.887242] Key type .llcrypt registered [ 1149.772332] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:51:54 (1785390714) [ 1151.039169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1151.095056] alg: No test for adler32 (adler32-zlib) [ 1152.373162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1152.497683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1152.563221] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1154.166990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1154.197222] LNet: 43482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.205919] LNetError: 43482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1154.232612] LNet: Removed LNI 192.168.204.29@tcp [ 1154.281959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1154.432816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 1154.492561] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1167.664746] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:52:12 (1785390732) [ 1167.838044] LNet: 44068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1167.850858] LNetError: 44068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1167.884667] LNet: Removed LNI 192.168.204.29@tcp [ 1168.766666] Key type .llcrypt unregistered [ 1168.768378] Key type ._llcrypt unregistered [ 1177.744172] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1177.886861] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1177.894589] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1177.912390] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1180.699606] Key type ._llcrypt registered [ 1180.701864] Key type .llcrypt registered [ 1180.851492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1180.869468] alg: No test for adler32 (adler32-zlib) [ 1181.869062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1181.916618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1181.936628] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1182.468484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1182.481860] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1185.355300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1185.394088] LNet: 44818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1185.401283] LNetError: 44818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1185.413812] LNet: Removed LNI 10.1.2.3@tcp [ 1185.428696] LNet: Removed LNI 192.168.204.29@tcp [ 1185.466469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1185.598834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1185.715075] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1185.719617] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1186.429699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1186.480835] LNet: 44846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.492246] LNetError: 44846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1186.502155] LNetError: 44846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1186.519328] LNet: Removed LNI 10.1.2.3@tcp [ 1186.547822] LNet: Removed LNI 192.168.204.29@tcp [ 1186.579337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1186.704399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1186.752107] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1186.761117] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1187.593425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1187.632409] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 1188.884888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1188.914401] LNet: 45008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1188.925473] LNetError: 45008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1188.932860] LNetError: 45008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1188.944341] LNet: Removed LNI 10.1.2.3@tcp [ 1188.948968] LNet: Removed LNI 192.168.204.29@tcp [ 1188.971359] LNet: Removed LNI 192.168.204.29@tcp2 [ 1189.015925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1189.096311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1189.138596] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1189.143300] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1189.150434] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 1200.602567] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:52:45 (1785390765) [ 1200.766773] LNet: 45603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.775843] LNetError: 45603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1200.780926] LNetError: 45603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1200.784259] LNet: Removed LNI 10.1.2.3@tcp [ 1200.790868] LNet: Removed LNI 192.168.204.29@tcp [ 1200.824175] LNet: Removed LNI 192.168.204.29@tcp2 [ 1201.835872] Key type .llcrypt unregistered [ 1201.842846] Key type ._llcrypt unregistered [ 1212.611409] Key type ._llcrypt registered [ 1212.617831] Key type .llcrypt registered [ 1212.941077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1212.965497] alg: No test for adler32 (adler32-zlib) [ 1214.179760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1214.341238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1214.399043] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1216.161948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1216.214139] LNet: 46005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.229698] LNetError: 46005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.260702] LNet: Removed LNI 192.168.204.29@tcp [ 1216.297680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1216.470255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1216.552249] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1217.366775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1217.397035] LNet: 46036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1217.416589] LNetError: 46036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1217.436414] LNet: Removed LNI 192.168.204.29@tcp [ 1217.495479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1217.630912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1217.715467] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1218.999580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1219.045592] LNet: 46066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1219.061559] LNetError: 46066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1219.109222] LNet: Removed LNI 192.168.204.29@tcp [ 1219.160229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1219.302938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1219.372587] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1220.199917] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1220.487664] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1220.496951] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1220.507447] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1223.021722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1223.073560] LNet: 46144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.090626] LNetError: 46144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1223.115985] LNet: Removed LNI 192.168.204.29@tcp [ 1223.170715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1223.266432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1223.317568] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1232.708468] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:53:18 (1785390798) [ 1233.033469] LNet: 46743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1233.038573] LNetError: 46743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1233.068840] LNet: Removed LNI 192.168.204.29@tcp [ 1234.036538] Key type .llcrypt unregistered [ 1234.042275] Key type ._llcrypt unregistered [ 1242.725640] Key type ._llcrypt registered [ 1242.732349] Key type .llcrypt registered [ 1242.950884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1242.972941] alg: No test for adler32 (adler32-zlib) [ 1244.018775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1244.153942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1244.195545] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1245.563616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1245.588595] LNet: 47142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1245.593637] LNetError: 47142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1245.618826] LNet: Removed LNI 192.168.204.29@tcp [ 1245.658922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1245.732681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1245.774310] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1256.393733] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:53:40 (1785390820) [ 1256.576487] LNet: 47727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1256.587904] LNetError: 47727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1256.641289] LNet: Removed LNI 192.168.204.29@tcp [ 1257.933438] Key type .llcrypt unregistered [ 1257.938235] Key type ._llcrypt unregistered [ 1268.985081] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1269.253519] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1269.263386] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1269.277637] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1272.593400] Key type ._llcrypt registered [ 1272.601143] Key type .llcrypt registered [ 1273.135411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1273.179294] alg: No test for adler32 (adler32-zlib) [ 1274.514719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1274.721918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1274.974518] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1277.129957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1277.182899] LNet: 48186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1277.214917] LNetError: 48186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.240138] LNet: Removed LNI 192.168.204.29@tcp [ 1277.365317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1277.568155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1277.611516] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1290.853849] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:54:15 (1785390855) [ 1291.120600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1291.184168] LNet: 48785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1291.195101] LNetError: 48785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1291.226824] LNet: Removed LNI 192.168.204.29@tcp [ 1291.279480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1291.455229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1300.973547] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:54:25 (1785390865) [ 1301.081610] LNet: 49360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.776488] Key type .llcrypt unregistered [ 1301.780736] Key type ._llcrypt unregistered [ 1310.090672] Key type ._llcrypt registered [ 1310.092176] Key type .llcrypt registered [ 1310.547930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1310.614331] alg: No test for adler32 (adler32-zlib) [ 1311.703655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1311.854517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1311.922783] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1313.748106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1313.814993] LNet: 49766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.834970] LNetError: 49766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.874157] LNet: Removed LNI 192.168.204.29@tcp [ 1313.920692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1314.109243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1314.185339] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1323.986974] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:54:49 (1785390889) [ 1324.269732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1324.297480] LNet: 50360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.303587] LNetError: 50360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.327424] LNet: Removed LNI 192.168.204.29@tcp [ 1324.362771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.433707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1333.194363] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:54:58 (1785390898) [ 1335.227365] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1337.658264] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:55:02 (1785390902) [ 1340.100343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1342.336070] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:55:07 (1785390907) [ 1345.277895] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1347.743197] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:55:12 (1785390912) [ 1349.969509] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1352.427215] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:55:17 (1785390917) [ 1353.810940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1356.098234] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:55:21 (1785390921) [ 1356.329561] LNet: 52410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.435697] Key type .llcrypt unregistered [ 1357.440455] Key type ._llcrypt unregistered [ 1366.082561] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1366.382062] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1366.394894] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1366.403069] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1369.220498] Key type ._llcrypt registered [ 1369.222203] Key type .llcrypt registered [ 1369.535779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1369.559056] alg: No test for adler32 (adler32-zlib) [ 1370.553568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1370.665694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1370.700142] LNet: Added LNI 192.168.204.29@tcp [8/512/0/180] [ 1370.709949] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1370.722720] LNet: 52734:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1370.906399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1370.960305] LNet: 52753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.980843] LNetError: 52753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1370.993146] LNet: Removed LNI 10.1.2.3@tcp [ 1370.997689] LNetError: 52753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.014644] LNet: Removed LNI 192.168.204.29@tcp [ 1371.042796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1371.135983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1371.167937] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1371.287982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1371.316549] LNet: 52777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1371.328229] LNetError: 52777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.343677] LNet: Removed LNI 192.168.204.29@tcp [ 1371.364840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1371.446197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1371.490674] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1371.497123] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1371.500463] LNet: 52785:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1371.614698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1371.651767] LNet: 52803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1371.665818] LNetError: 52803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1371.668513] LNet: Removed LNI 10.1.2.3@tcp [ 1371.677346] LNetError: 52803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.696714] LNet: Removed LNI 192.168.204.29@tcp [ 1371.734161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1371.821246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1371.869741] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1371.895764] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1371.897557] LNet: 52811:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1372.245343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1372.278117] LNet: 52837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.286385] LNetError: 52837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1372.291061] LNet: Removed LNI 10.1.2.3@tcp [ 1372.298445] LNetError: 52837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.324959] LNet: Removed LNI 192.168.204.29@tcp [ 1372.373604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1372.494338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1372.545175] LNet: Added LNI 192.168.204.29@tcp [8/256/0/182] [ 1372.558489] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1372.737960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1372.782210] LNet: 52865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.790678] LNetError: 52865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1372.802673] LNet: Removed LNI 10.1.2.3@tcp [ 1372.806866] LNetError: 52865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.827654] LNet: Removed LNI 192.168.204.29@tcp [ 1372.862081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1372.991172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1373.068160] LNet: Added LNI 192.168.204.29@tcp [8/256/0/182] [ 1373.081144] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1373.085612] LNet: 52873:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1373.247796] LNet: Removed LNI 10.1.2.3@tcp [ 1373.263177] LNet: Removed LNI 192.168.204.29@tcp [ 1374.198136] Key type .llcrypt unregistered [ 1374.206917] Key type ._llcrypt unregistered [ 1389.301725] Key type ._llcrypt registered [ 1389.306833] Key type .llcrypt registered [ 1391.595493] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:55:56 (1785390956) [ 1392.129770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1392.160932] alg: No test for adler32 (adler32-zlib) [ 1393.264189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1393.333509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1393.373258] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1394.316528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.29 [ 1394.368678] LNet: 53851:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1394.382479] LNet: Removed LNI 192.168.204.29@tcp [ 1403.696468] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:56:08 (1785390968) [ 1403.890188] LNet: 54414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.907229] LNetError: 54414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1403.964559] LNet: Removed LNI 192.168.204.29@tcp [ 1405.085537] Key type .llcrypt unregistered [ 1405.089225] Key type ._llcrypt unregistered [ 1414.487151] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1414.829237] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1418.127525] Key type ._llcrypt registered [ 1418.133180] Key type .llcrypt registered [ 1418.555780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1418.585905] alg: No test for adler32 (adler32-zlib) [ 1420.482244] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1428.970328] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:56:34 (1785390994) [ 1429.043240] LNet: 55327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1429.050142] LNetError: 55327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1429.083880] LNet: Removed LNI 10.1.2.3@tcp [ 1429.697519] Key type .llcrypt unregistered [ 1429.702435] Key type ._llcrypt unregistered [ 1439.165101] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1439.617540] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1439.628684] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1439.637880] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1443.076584] Key type ._llcrypt registered [ 1443.082202] Key type .llcrypt registered [ 1443.519306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1443.551501] alg: No test for adler32 (adler32-zlib) [ 1444.775530] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1456.519074] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:57:00 (1785391020) [ 1459.648614] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1462.095955] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:57:07 (1785391027) [ 1463.020385] LNet: 56527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.060315] LNetError: 56527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1463.105915] LNet: Removed LNI 10.1.2.3@tcp [ 1464.129346] Key type .llcrypt unregistered [ 1464.133382] Key type ._llcrypt unregistered [ 1473.581749] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1473.930207] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1473.942775] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1473.952736] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1476.848237] Key type ._llcrypt registered [ 1476.852856] Key type .llcrypt registered [ 1477.075631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1477.094452] alg: No test for adler32 (adler32-zlib) [ 1478.154728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.272988] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1490.939537] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:57:36 (1785391056) [ 1491.845316] LNet: 57759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1491.866395] LNetError: 57759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1491.902142] LNet: Removed LNI 10.1.2.3@tcp [ 1493.035454] Key type .llcrypt unregistered [ 1493.036937] Key type ._llcrypt unregistered [ 1505.411251] Key type ._llcrypt registered [ 1505.416096] Key type .llcrypt registered [ 1505.798460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1505.878275] alg: No test for adler32 (adler32-zlib) [ 1506.977585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1507.072530] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1520.034259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1521.092461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_interrupt -r 1 [ 1521.166774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_interrupt -r 1 [ 1521.221940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1521.241343] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1521.444737] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 1521.539987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1521.569946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1521.636960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1521.731984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1521.798780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1521.882106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1521.951277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1521.992796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1522.020209] LNet: 58199:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1522.040107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1522.082863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1522.123252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1522.158184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1522.196992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1522.224426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1522.268336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1522.965427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_dropped -r 1 [ 1523.027654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_dropped -r 1 [ 1523.120648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1523.164384] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1523.507854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1523.653396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1523.740756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1523.812912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1523.870915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1523.948265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1524.006463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1524.063419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1524.096768] LNet: 58309:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1524.127821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1524.177986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1524.259836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1524.366959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1524.450411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1524.515542] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1524.529700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1524.608779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1525.313991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_aborted -r 1 [ 1525.361941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_aborted -r 1 [ 1525.417967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1525.694986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1525.739126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1525.799344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1525.854986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1525.902965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1525.931871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1525.955662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1525.987929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1526.007519] LNet: 58418:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1526.023426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1526.053189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1526.077777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1526.100861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1526.125529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1526.156263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1526.180937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1526.580894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_no_route -r 1 [ 1526.608256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_no_route -r 1 [ 1526.644577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1526.668552] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1526.673620] LNet: Skipped 1 previous similar message [ 1526.905487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1526.945682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1526.993676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1527.053313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1527.101661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1527.163238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1527.217405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1527.278300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1527.320978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1527.362390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1527.407406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1527.427635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1527.452701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1527.477705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1527.511512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1527.975384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_timeout -r 1 [ 1528.021889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_timeout -r 1 [ 1528.049571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1528.302742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1528.343525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1528.385973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1528.406269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1528.438513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1528.457116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1528.478899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1528.504716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1528.519431] LNet: 58636:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1528.525480] LNet: 58636:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1528.551189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1528.581280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1528.605523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1528.637456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1528.671538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1528.700505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1528.750175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1529.319382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_error -r 1 [ 1529.383402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_error -r 1 [ 1529.437077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1529.796922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1529.850517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1529.903782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1529.948969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1529.986785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1530.027939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1530.065079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1530.105582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1530.141501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1530.169821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1530.200892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1530.233939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1530.267470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1530.298482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1530.353249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1530.630547] LNet: 58787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.640246] LNetError: 58787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1531.695333] LNet: Removed LNI 192.168.204.29@tcp [ 1532.350681] Key type .llcrypt unregistered [ 1532.359212] Key type ._llcrypt unregistered [ 1557.700727] Key type ._llcrypt registered [ 1557.704948] Key type .llcrypt registered [ 1559.766451] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:58:44 (1785391124) [ 1561.420261] Key type .llcrypt unregistered [ 1561.425201] Key type ._llcrypt unregistered [ 1569.782927] Key type ._llcrypt registered [ 1569.787904] Key type .llcrypt registered [ 1570.273984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1570.320668] alg: No test for adler32 (adler32-zlib) [ 1571.681371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1571.758711] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1587.593829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1604.475918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.537791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1604.576587] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1606.741900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_interrupt -r 1 [ 1606.778174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_interrupt -r 1 [ 1606.819283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_interrupt -r 1 [ 1606.884392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_interrupt -r 1 [ 1606.936740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_interrupt -r 1 [ 1606.996593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_interrupt -r 1 [ 1607.051366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_interrupt -r 1 [ 1607.103662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_interrupt -r 1 [ 1607.154742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1607.183978] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1607.189633] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 1607.509440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1607.531240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1607.553390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1607.574842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1607.598189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1607.625097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1607.658738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1607.692129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1607.710633] LNet: 60260:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1607.729952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1607.762902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1607.806143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1607.835330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1607.860945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1607.899395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1607.954463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1608.636873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_dropped -r 1 [ 1608.680367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_dropped -r 1 [ 1608.713686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_dropped -r 1 [ 1608.753222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_dropped -r 1 [ 1608.785421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_dropped -r 1 [ 1608.829530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_dropped -r 1 [ 1608.872974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_dropped -r 1 [ 1608.925701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_dropped -r 1 [ 1608.971061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1608.991454] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1608.998231] LNet: Skipped 2 previous similar messages [ 1609.218904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1609.243249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1609.275809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1609.322447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1609.352353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1609.382266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1609.410708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1609.442548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1609.463259] LNet: 60382:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1609.487384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1609.521728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1609.566459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1609.611713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1609.676940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1609.762286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1609.807528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1610.601947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_aborted -r 1 [ 1610.665391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_aborted -r 1 [ 1610.733099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_aborted -r 1 [ 1610.817341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_aborted -r 1 [ 1610.891952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_aborted -r 1 [ 1610.956267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_aborted -r 1 [ 1611.016866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_aborted -r 1 [ 1611.080291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_aborted -r 1 [ 1611.122696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1611.137227] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1611.142213] LNet: Skipped 2 previous similar messages [ 1611.399965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1611.431766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1611.498900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1611.546510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1611.584463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1611.659420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1611.701453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1611.759382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1611.782070] LNet: 60504:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1611.818450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1611.870600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1611.937810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1611.986987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1612.067125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1612.146485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1612.227390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1613.048359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_no_route -r 1 [ 1613.094557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_no_route -r 1 [ 1613.144862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_no_route -r 1 [ 1613.209632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_no_route -r 1 [ 1613.268853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_no_route -r 1 [ 1613.337874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_no_route -r 1 [ 1613.406610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_no_route -r 1 [ 1613.494519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_no_route -r 1 [ 1613.556203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1613.593793] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1613.600813] LNet: Skipped 2 previous similar messages [ 1613.857689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1613.886979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1613.916572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1613.941453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1613.963884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1613.991881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1614.022209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1614.062409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1614.085880] LNet: 60625:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1614.098327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1614.124191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1614.146848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1614.172250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1614.194483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1614.239533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1614.273477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1614.706851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_timeout -r 1 [ 1614.735350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_timeout -r 1 [ 1614.769428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_timeout -r 1 [ 1614.828818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_timeout -r 1 [ 1614.890324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_timeout -r 1 [ 1614.948426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_timeout -r 1 [ 1615.017823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_timeout -r 1 [ 1615.080993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_timeout -r 1 [ 1615.142862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1615.472775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1615.519285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1615.569492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1615.614702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1615.658509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1615.710872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1615.759368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1615.823844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1615.881355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1615.924359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1615.958166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1615.992194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1616.043417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1616.094781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1616.153842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1616.926099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_error -r 1 [ 1616.997451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_error -r 1 [ 1617.074224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_error -r 1 [ 1617.142283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_error -r 1 [ 1617.208457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_error -r 1 [ 1617.271237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_error -r 1 [ 1617.332977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_error -r 1 [ 1617.375842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_error -r 1 [ 1617.447276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1617.807798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1617.859496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1617.954874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1618.048250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1618.106919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1618.153374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1618.193812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1618.246972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1618.274112] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1618.281556] LNet: 60868:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1618.284200] LNet: Skipped 6 previous similar messages [ 1618.297196] LNet: 60868:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1618.316646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1618.371223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1618.414575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1618.448665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1618.489510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1618.548454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1618.600692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1626.092573] LNet: 60944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.127644] LNetError: 60944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.169400] LNet: Removed LNI 192.168.204.29@tcp [ 1627.178804] LNetError: 60944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.204420] LNet: Removed LNI 192.168.204.29@tcp1 [ 1627.934838] Key type .llcrypt unregistered [ 1627.936436] Key type ._llcrypt unregistered [ 1653.840523] Key type ._llcrypt registered [ 1653.841844] Key type .llcrypt registered [ 1656.009477] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 02:00:20 (1785391220) [ 1657.334940] Key type .llcrypt unregistered [ 1657.338093] Key type ._llcrypt unregistered [ 1667.764260] Key type ._llcrypt registered [ 1667.766677] Key type .llcrypt registered [ 1668.204370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1668.239286] alg: No test for adler32 (adler32-zlib) [ 1669.378567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1669.423938] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1678.259533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1679.443390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_dropped -r 1 [ 1679.492454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_dropped -r 1 [ 1679.550288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1679.584938] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1679.989499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1680.034701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1680.078742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1680.134844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1680.184111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1680.239192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1680.290863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1680.338927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1680.362792] LNet: 62197:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1680.387560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1680.428630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1680.461133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1680.506131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1680.551859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1680.591555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1680.668391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1681.450729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_error -r 1 [ 1681.513716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_error -r 1 [ 1681.594061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1681.624348] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1682.148410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1682.263416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1682.352726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1682.517695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1682.620042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1682.723571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1682.892405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1682.985209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1683.057412] LNet: 62307:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1683.147319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1683.293437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1683.449133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1683.578841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1683.695782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1683.782902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1683.853869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1684.843801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_timeout -r 1 [ 1684.899041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_timeout -r 1 [ 1684.980926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1685.018076] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1685.414142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1685.494930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1685.552993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1685.627601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1685.686518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1685.810835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1685.887248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1685.977133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1686.024222] LNet: 62416:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1686.071782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1686.119883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1686.173663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1686.207589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1686.246584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1686.267749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1686.312735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1686.659884] LNet: 62457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.669994] LNetError: 62457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.690564] LNet: Removed LNI 192.168.204.29@tcp [ 1687.280926] Key type .llcrypt unregistered [ 1687.285129] Key type ._llcrypt unregistered [ 1715.070920] Key type ._llcrypt registered [ 1715.077351] Key type .llcrypt registered [ 1717.008442] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 02:01:21 (1785391281) [ 1718.025435] Key type .llcrypt unregistered [ 1718.032279] Key type ._llcrypt unregistered [ 1729.854869] Key type ._llcrypt registered [ 1729.859143] Key type .llcrypt registered [ 1730.234409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1730.261445] alg: No test for adler32 (adler32-zlib) [ 1731.256745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1731.287559] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1745.295922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1758.908479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.983463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1759.020431] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1761.879505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_dropped -r 1 [ 1762.014744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e remote_dropped -r 1 [ 1762.116520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_dropped -r 1 [ 1762.210203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e remote_dropped -r 1 [ 1762.329619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e remote_dropped -r 1 [ 1762.419465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e remote_dropped -r 1 [ 1762.566495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e remote_dropped -r 1 [ 1762.705425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e remote_dropped -r 1 [ 1762.755583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1762.811803] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1762.820639] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.129@tcp [ 1763.328124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1763.389497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1763.451874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1763.506661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1763.559888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1763.639277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1763.699052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1763.750852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1763.777357] LNet: 63944:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1763.815207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1763.879449] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1763.894066] LNet: Skipped 3 previous similar messages [ 1763.894358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1763.957799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1764.026591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1764.117750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1764.205966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1764.290237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1764.359350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1765.454894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_error -r 1 [ 1765.575923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e remote_error -r 1 [ 1765.650457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_error -r 1 [ 1765.749746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e remote_error -r 1 [ 1765.807413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e remote_error -r 1 [ 1765.858636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e remote_error -r 1 [ 1765.905349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e remote_error -r 1 [ 1765.980751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e remote_error -r 1 [ 1766.037822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1766.065948] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1766.538869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1766.591366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1766.650421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1766.695825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1766.757357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1766.833041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1766.909671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1766.961635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1767.012704] LNet: 64068:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1767.058797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1767.114477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1767.151218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1767.197283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1767.235738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1767.261183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1767.285859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1767.314557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1768.099531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_timeout -r 1 [ 1768.176370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e remote_timeout -r 1 [ 1768.255504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_timeout -r 1 [ 1768.319857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e remote_timeout -r 1 [ 1768.384777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e remote_timeout -r 1 [ 1768.410213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e remote_timeout -r 1 [ 1768.433828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e remote_timeout -r 1 [ 1768.457517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e remote_timeout -r 1 [ 1768.484118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1768.507788] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1768.833363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1768.888891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1768.933782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1768.968787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1769.026545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1769.087196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1769.171438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1769.225339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1769.255491] LNet: 64191:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1769.290194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1769.344395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1769.397712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1769.474570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1769.544438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1769.612797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1769.694666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1769.760969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1777.338475] LNet: 64269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.347877] LNetError: 64269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.358808] LNet: Removed LNI 192.168.204.29@tcp [ 1777.379905] LNet: Removed LNI 192.168.204.29@tcp1 [ 1778.017496] Key type .llcrypt unregistered [ 1778.019871] Key type ._llcrypt unregistered [ 1806.861538] Key type ._llcrypt registered [ 1806.869042] Key type .llcrypt registered [ 1809.392161] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 02:02:53 (1785391373) [ 1811.527673] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1814.621525] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 02:02:59 (1785391379) [ 1815.833288] Key type .llcrypt unregistered [ 1815.837692] Key type ._llcrypt unregistered [ 1827.496737] Key type ._llcrypt registered [ 1827.498258] Key type .llcrypt registered [ 1827.958099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1828.008136] alg: No test for adler32 (adler32-zlib) [ 1829.093519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1829.152733] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1839.732160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1840.835324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e network_timeout -r 1 [ 1840.895758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e network_timeout -r 1 [ 1840.939390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1840.970204] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1841.268660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1841.330533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1841.372464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1841.415639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1841.442743] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 1841.477787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1841.536344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1841.580655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1841.618314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1841.640295] LNet: 65819:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1841.675583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1841.726127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1841.772028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1841.810966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1841.875940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1841.946614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1842.403266] LNet: 65858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.409584] LNetError: 65858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1842.435860] LNet: Removed LNI 192.168.204.29@tcp [ 1843.026463] Key type .llcrypt unregistered [ 1843.034077] Key type ._llcrypt unregistered [ 1871.047663] Key type ._llcrypt registered [ 1871.057810] Key type .llcrypt registered [ 1871.416474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1871.482565] alg: No test for adler32 (adler32-zlib) [ 1872.712472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1872.809956] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1885.536628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1899.235634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1899.257558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1899.268797] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1900.898254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e network_timeout -r 1 [ 1900.954544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e network_timeout -r 1 [ 1900.984741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e network_timeout -r 1 [ 1901.043251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e network_timeout -r 1 [ 1901.093725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e network_timeout -r 1 [ 1901.144431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e network_timeout -r 1 [ 1901.200647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e network_timeout -r 1 [ 1901.236604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e network_timeout -r 1 [ 1901.305568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1901.338425] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1901.538430] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 1901.546032] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.129@tcp [ 1901.701121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1901.721665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1901.751842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1901.788948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1901.822450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1901.847759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1901.879536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1901.908272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1901.935717] LNet: 66741:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1901.951995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1901.983715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1902.015355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1902.049601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1902.085802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1902.130394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1907.682194] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1908.433595] LNet: 66816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.451324] LNetError: 66816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1908.463808] LNet: Removed LNI 192.168.204.29@tcp [ 1908.490794] LNet: Removed LNI 192.168.204.29@tcp1 [ 1909.303430] Key type .llcrypt unregistered [ 1909.305691] Key type ._llcrypt unregistered [ 1935.409438] Key type ._llcrypt registered [ 1935.412412] Key type .llcrypt registered [ 1937.447658] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 02:05:02 (1785391502) [ 1938.121191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1938.148368] alg: No test for adler32 (adler32-zlib) [ 1939.225440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1939.348781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1939.413522] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1940.458640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1940.497783] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1941.837697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 1941.939355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1942.008443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1942.092794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1942.144271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1942.174483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1942.239535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.29@tcp [ 1942.277450] LNetError: 67918:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.29@tcp to recovery [ 1942.372442] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 1942.378778] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1944.416431] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1945.088163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1948.517321] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1949.391194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1949.450190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1949.583475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1949.603719] LNet: 68016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1949.613722] LNetError: 68016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1949.622746] LNet: Removed LNI 192.168.204.29@tcp [ 1949.645179] LNet: Removed LNI 192.168.204.29@tcp1 [ 1949.755063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.866504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1949.916665] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1951.447281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1951.479129] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1953.206894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 1953.274242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1953.373992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1953.462394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1953.511114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1953.587770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1953.629509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1953.699170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.29@tcp [ 1953.750500] LNetError: 68288:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.29@tcp to recovery [ 1953.892475] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1956.465258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1959.009368] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1959.014892] LNet: Skipped 1 previous similar message [ 1963.667063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1963.709250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1963.774191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1974.454386] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 02:05:39 (1785391539) [ 1974.557393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1974.592384] LNet: 68981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1974.604403] LNetError: 68981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1974.618269] LNetError: 68981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1974.655256] LNet: Removed LNI 192.168.204.29@tcp [ 1974.680766] LNet: Removed LNI 192.168.204.29@tcp1 [ 1974.734240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1974.822169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1974.851470] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1975.785086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1975.816302] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1976.910704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 1976.968670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1976.998515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1977.032427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1977.073960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1977.121483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1977.168735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.29@tcp --health 0 [ 1977.313242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1977.826284] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.29@tcp [ 1978.848566] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1978.855453] LNet: Skipped 1 previous similar message [ 1992.662284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1993.023077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1993.068424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1993.116650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.29@tcp --health 500 [ 1993.241247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2000.377495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2000.496899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2000.536427] LNet: 69472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.547872] LNetError: 69472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.557660] LNetError: 69472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2000.568960] LNet: Removed LNI 192.168.204.29@tcp [ 2000.594841] LNet: Removed LNI 192.168.204.29@tcp1 [ 2000.642886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2000.730001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2000.786077] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2001.645562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2001.679575] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 2002.781736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2002.832161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2002.912782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 2002.956788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 2003.017435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 2003.109671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2003.174602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 2003.232754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.29@tcp --health 0 [ 2004.770598] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 2004.778443] LNet: Skipped 3 previous similar messages [ 2005.912121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2015.819586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2015.881388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2015.949545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2028.142770] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 02:06:32 (1785391592) [ 2028.680709] LNet: 70519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.687718] LNetError: 70519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.692363] LNetError: 70519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2028.696758] LNet: Removed LNI 192.168.204.29@tcp [ 2028.707117] LNet: Removed LNI 192.168.204.29@tcp1 [ 2029.600555] Key type .llcrypt unregistered [ 2029.605084] Key type ._llcrypt unregistered [ 2038.959469] Key type ._llcrypt registered [ 2038.967559] Key type .llcrypt registered [ 2039.318468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2039.345839] alg: No test for adler32 (adler32-zlib) [ 2040.314497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2040.344936] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2058.166297] Lustre: DEBUG MARKER: Initial discovery [ 2058.497638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.129@tcp [ 2063.409593] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 2063.504825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 2063.590620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.129@tcp [ 2063.644309] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 2063.693542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2063.774467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.129@tcp -r 1 -m GET -l 3 [ 2063.857630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.129@tcp -r 1 -m GET -e local_error [ 2063.904289] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 2063.953250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.129@tcp -r 1 -m PUT -l 6 [ 2066.640264] Lustre: DEBUG MARKER: Force oleg429-client.virtnet to discover 192.168.204.129@tcp (in background) [ 2067.201598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.129@tcp [ 2069.188185] Lustre: DEBUG MARKER: Force oleg429-server to discover 192.168.204.29@tcp [ 2070.306726] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 2072.897751] Lustre: DEBUG MARKER: Wait for 71288 [ 2074.088476] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 2075.308281] Lustre: DEBUG MARKER: Finished wait on 71288 [ 2075.393728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 2075.472157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2076.000115] LNet: 71736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.026369] LNetError: 71736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.048770] LNet: Removed LNI 192.168.204.29@tcp [ 2076.768433] Key type .llcrypt unregistered [ 2076.772813] Key type ._llcrypt unregistered [ 2099.942661] Key type ._llcrypt registered [ 2099.945899] Key type .llcrypt registered [ 2101.851105] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 02:07:46 (1785391666) [ 2102.452280] Key type .llcrypt unregistered [ 2102.458846] Key type ._llcrypt unregistered [ 2111.102371] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2111.260828] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2111.278244] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2111.292626] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2114.094532] Key type ._llcrypt registered [ 2114.102019] Key type .llcrypt registered [ 2114.467584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2114.505738] alg: No test for adler32 (adler32-zlib) [ 2115.572651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2115.659508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2115.691285] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2116.780800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2116.807309] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2131.411430] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 02:08:15 (1785391695) [ 2131.550912] LNet: 73809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.558964] LNetError: 73809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2131.571782] LNet: Removed LNI 10.1.2.3@tcp [ 2131.613681] LNet: Removed LNI 192.168.204.29@tcp [ 2132.303259] Key type .llcrypt unregistered [ 2132.308484] Key type ._llcrypt unregistered [ 2141.334613] Key type ._llcrypt registered [ 2141.336539] Key type .llcrypt registered [ 2141.711561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2141.735377] alg: No test for adler32 (adler32-zlib) [ 2142.639937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2142.711639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2142.732968] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2143.621759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2143.643969] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2144.494502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.29@tcp [ 2145.570383] LNetError: Unexpected error -22 connecting to 192.168.204.29@tcp at host 192.168.204.29:988 [ 2153.646456] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 02:08:38 (1785391718) [ 2153.868334] LNet: 74947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2153.885072] LNetError: 74947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2153.900779] LNet: Removed LNI 10.1.2.3@tcp [ 2153.918846] LNet: Removed LNI 192.168.204.29@tcp [ 2155.043877] Key type .llcrypt unregistered [ 2155.046678] Key type ._llcrypt unregistered [ 2165.976653] Key type ._llcrypt registered [ 2165.978877] Key type .llcrypt registered [ 2166.340589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2166.356467] alg: No test for adler32 (adler32-zlib) [ 2167.513967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2167.631969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2167.764403] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 2169.451881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2169.515490] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 2170.412441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.29@tcp1 --nid 192.168.204.29@tcp2 [ 2176.697555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp1 192.168.204.29@tcp2 [ 2176.756953] LNetError: 75790:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.29@tcp1 to 192.168.204.29@tcp2 - Route table may be misconfigured [ 2176.821383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp2 192.168.204.29@tcp1 [ 2193.667752] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 02:09:17 (1785391757) [ 2193.860799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2193.954545] LNet: 76361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.970667] LNetError: 76361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2194.976098] LNet: Removed LNI 192.168.204.29@tcp1 [ 2194.985473] LNetError: 76361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2196.025455] LNet: Removed LNI 192.168.204.29@tcp2 [ 2196.145894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2196.275737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2196.345439] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2197.769367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2197.788623] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 2198.538304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2198.615860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.29@tcp [ 2198.680183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp1 192.168.204.29@tcp1 [ 2198.774068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.29@tcp -d 192.168.204.29@tcp -e network_timeout [ 2198.828808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -e network_timeout [ 2198.899194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -e network_timeout [ 2198.943185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -e network_timeout [ 2198.994967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2199.043315] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 2199.091676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2199.264131] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 2199.335852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2199.514855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2209.053265] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 02:09:33 (1785391773) [ 2209.213733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2209.282318] LNet: 77223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.318446] LNetError: 77223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2209.350238] LNet: Removed LNI 192.168.204.29@tcp [ 2209.383987] LNet: Removed LNI 192.168.204.29@tcp1 [ 2209.465538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.665596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 2210.384090] LNet: 77269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2211.275593] Key type .llcrypt unregistered [ 2211.287120] Key type ._llcrypt unregistered [ 2233.514404] Key type ._llcrypt registered [ 2233.516213] Key type .llcrypt registered [ 2234.940956] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 02:10:00 (1785391800) [ 2235.851752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2235.896442] alg: No test for adler32 (adler32-zlib) [ 2236.899538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.338906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 2241.507629] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 02:10:06 (1785391806) [ 2241.729481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2241.750420] LNet: 78370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2241.775678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.867775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2241.898713] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2243.381390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2243.417043] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 2245.423960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2245.527936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp1 [ 2245.619732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp1 [ 2253.966530] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 02:10:19 (1785391819) [ 2256.364925] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg429-server [ 2258.611693] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 02:10:24 (1785391824) [ 2261.397741] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg429-server [ 2263.588368] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 02:10:29 (1785391829) [ 2265.786199] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg429-server [ 2268.230397] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 02:10:33 (1785391833) [ 2269.992856] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg429-server [ 2272.670429] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 02:10:37 (1785391837) [ 2275.465600] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg429-server [ 2278.250844] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 02:10:43 (1785391843) [ 2281.060126] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg429-server [ 2283.318323] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 02:10:48 (1785391848) [ 2285.505782] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg429-server [ 2288.307530] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 02:10:53 (1785391853) [ 2291.141116] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg429-server [ 2293.718173] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 02:10:58 (1785391858) [ 2295.900268] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg429-server [ 2298.106867] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 02:11:03 (1785391863) [ 2298.289936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2298.342321] LNet: 82247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2298.355461] LNetError: 82247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2298.368060] LNet: Removed LNI 192.168.204.29@tcp [ 2299.431949] LNet: Removed LNI 192.168.204.29@tcp1 [ 2299.481084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.589941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2299.631941] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2300.881738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 2301.075520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2301.357412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2301.409766] LNet: 82398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.421060] LNetError: 82398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2301.433196] LNetError: 82398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2301.463936] LNet: Removed LNI 192.168.204.29@tcp [ 2301.531159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.633664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 2301.674819] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2302.958972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2303.233788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2303.287194] LNetError: 82543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2304.361156] LNet: Removed LNI 192.168.204.29@tcp [ 2304.408254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.530393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2304.565423] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2305.392488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 2305.515830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2305.742695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2305.781621] LNet: 82691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.792201] LNet: 82691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2305.808041] LNetError: 82691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2305.832677] LNet: Removed LNI 192.168.204.29@tcp [ 2305.869587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2305.986972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 2306.049888] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2307.020867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2307.265657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2308.358167] LNet: Removed LNI 192.168.204.29@tcp [ 2308.397955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2308.493624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2308.532400] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2309.738495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 2309.801905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2311.133558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.170910] LNet: 82992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.177481] LNet: 82992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2311.189807] LNetError: 82992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2311.195117] LNetError: 82992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2311.209989] LNet: Removed LNI 192.168.204.29@tcp [ 2311.231600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.316491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 2311.360332] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2312.562477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2312.853820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2313.896688] LNet: Removed LNI 192.168.204.29@tcp [ 2313.939277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2314.017330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2314.051198] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2314.716333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 2314.806643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2315.043256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2316.138571] LNet: Removed LNI 192.168.204.29@tcp [ 2316.248225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.414837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 2316.483093] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2318.113737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2318.498904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2319.591636] LNet: Removed LNI 192.168.204.29@tcp [ 2319.641557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.745151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2319.791082] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2320.900826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 2321.027594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2321.459170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2321.608450] LNet: 83578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2321.623453] LNet: 83578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2321.647855] LNetError: 83578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2321.650348] LNetError: 83578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2321.695066] LNet: Removed LNI 192.168.204.29@tcp [ 2321.790153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2321.946223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 2322.015579] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2323.492609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2325.159717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2326.247913] LNet: Removed LNI 192.168.204.29@tcp [ 2326.308815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.410936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2326.454763] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2327.950536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 2328.068127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2328.531768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2329.595204] LNet: Removed LNI 192.168.204.29@tcp [ 2329.723913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2329.821387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 2329.883060] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2331.031081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2333.012312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.133338] LNet: Removed LNI 192.168.204.29@tcp [ 2334.208329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.374632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2334.498768] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2336.527272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 2336.729547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2336.805675] LNet: Removed LNI 192.168.204.29@tcp [ 2336.882437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.110655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2337.211237] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2339.195510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 2339.459905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.559891] LNet: 84305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.581686] LNet: 84305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2339.617797] LNetError: 84305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2339.644873] LNetError: 84305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2339.707124] LNet: Removed LNI 192.168.204.29@tcp [ 2339.789655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2339.990764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 2340.190597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.308834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.438314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 2340.618903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.738204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.905197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 2341.100658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2341.196910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2341.367579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 2341.451978] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2343.266105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.366457] LNet: Removed LNI 192.168.204.29@tcp [ 2343.418377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.613781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2357.217644] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 02:12:01 (1785391921) [ 2357.911809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.039573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.178786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 2358.237724] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/180] [ 2361.707195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2361.790860] LNet: Removed LNI 192.168.204.29@tcp231 [ 2361.865344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2362.069823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2362.113498] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/180] [ 2362.139399] LNet: 85204:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2363.676285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2363.758659] LNet: Removed LNI 192.168.204.29@tcp231 [ 2363.797694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 2363.902913] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/180] [ 2367.024718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2367.071713] LNet: Removed LNI 192.168.204.29@tcp231 [ 2367.127329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2367.294394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2367.370395] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/0] [ 2367.398407] LNet: 85357:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2368.314441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2368.360268] LNet: Removed LNI 192.168.204.29@tcp231 [ 2368.383805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 2368.427608] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/0] [ 2371.106613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2371.178255] LNet: Removed LNI 192.168.204.29@tcp231 [ 2371.207520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.329183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2371.377385] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/60] [ 2371.395591] LNet: 85508:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2371.954416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2372.034460] LNet: Removed LNI 192.168.204.29@tcp231 [ 2372.115697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 2372.232946] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/60] [ 2374.344165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2374.404224] LNet: 85652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.409332] LNet: 85652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 2374.415302] LNetError: 85652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2374.420319] LNetError: 85652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2374.449746] LNet: Removed LNI 192.168.204.29@tcp231 [ 2374.478344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2374.577994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2374.606287] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/180] [ 2374.610877] LNet: 85660:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2375.319746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2375.377078] LNet: Removed LNI 192.168.204.29@tcp231 [ 2375.406876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 2375.455569] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/180] [ 2386.553298] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 02:12:31 (1785391951) [ 2386.956590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2387.010452] LNet: Removed LNI 192.168.204.29@tcp231 [ 2387.055898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2387.168163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 2387.210562] LNet: Added LNI 192.168.204.29@tcp232 [8/256/0/180] [ 2388.150613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2388.194832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2388.242235] LNet: Removed LNI 192.168.204.29@tcp232 [ 2388.277821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 2388.336186] LNet: Added LNI 192.168.204.29@tcp232 [8/256/0/180] [ 2389.328391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2389.378114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2389.449587] LNet: Removed LNI 192.168.204.29@tcp232 [ 2389.464481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 2389.504075] LNet: Added LNI 192.168.204.29@tcp232 [8/256/0/180] [ 2390.328603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2390.396749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2390.434897] LNet: Removed LNI 192.168.204.29@tcp232 [ 2390.471812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 2390.519822] LNet: Added LNI 192.168.204.29@tcp232 [8/256/0/180] [ 2391.657846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2391.684368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2391.707714] LNet: Removed LNI 192.168.204.29@tcp232 [ 2391.725591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2400.472308] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 02:12:45 (1785391965) [ 2402.906925] Key type .llcrypt unregistered [ 2402.913863] Key type ._llcrypt unregistered [ 2415.374732] Key type ._llcrypt registered [ 2415.377844] Key type .llcrypt registered [ 2415.761814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2415.823726] alg: No test for adler32 (adler32-zlib) [ 2417.044243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2417.134072] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2429.815305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 2443.963744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2444.067857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2444.171778] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 2446.202739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_interrupt -r 2 [ 2446.302924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_interrupt -r 2 [ 2446.406898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_interrupt -r 2 [ 2446.490610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_interrupt -r 2 [ 2446.553471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_interrupt -r 2 [ 2446.625602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_interrupt -r 2 [ 2446.694992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_interrupt -r 2 [ 2446.779571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_interrupt -r 2 [ 2446.861893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 2446.900123] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 2446.916025] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 2447.272413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2447.313819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2447.361211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2447.401115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2447.455460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2447.537547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2447.596047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2447.670433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2447.715479] LNet: 88165:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2447.748281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2447.830136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2447.902762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2447.975100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2448.027317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2448.054899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2448.923749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_dropped -r 2 [ 2448.977611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_dropped -r 2 [ 2449.045633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_dropped -r 2 [ 2449.093887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_dropped -r 2 [ 2449.164777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_dropped -r 2 [ 2449.229248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_dropped -r 2 [ 2449.301648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_dropped -r 2 [ 2449.382599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_dropped -r 2 [ 2449.453912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 2449.493188] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 2449.913874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2449.963140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2450.012865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2450.076341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2450.147721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2450.224209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2450.288172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2450.347951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2450.400744] LNet: 88285:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2450.432497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2450.490023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2450.557906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2450.613584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2450.685295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2450.755749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2451.688576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_aborted -r 2 [ 2451.747137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_aborted -r 2 [ 2451.803923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_aborted -r 2 [ 2451.880924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_aborted -r 2 [ 2451.961855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_aborted -r 2 [ 2452.022753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_aborted -r 2 [ 2452.079583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_aborted -r 2 [ 2452.155618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_aborted -r 2 [ 2452.233753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 2452.275228] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 2452.639125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2452.689305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2452.745764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2452.797313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2452.856737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2452.915044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2452.962444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2453.060709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2453.099589] LNet: 88405:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2453.145351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2453.190855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2453.254589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2453.312814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2453.401495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2453.471139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2454.276939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_no_route -r 2 [ 2454.332445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_no_route -r 2 [ 2454.382792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_no_route -r 2 [ 2454.440556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_no_route -r 2 [ 2454.489082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_no_route -r 2 [ 2454.557946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_no_route -r 2 [ 2454.620791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_no_route -r 2 [ 2454.709764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_no_route -r 2 [ 2454.784177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 2454.814978] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 2455.142578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2455.191804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2455.239670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2455.285909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2455.330105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2455.371079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2455.425068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2455.460277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2455.486379] LNet: 88526:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2455.515236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2455.562391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2455.622081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2455.673843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2455.711758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2455.755848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2456.493853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_timeout -r 2 [ 2456.588413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_timeout -r 2 [ 2456.667548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_timeout -r 2 [ 2456.744909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_timeout -r 2 [ 2456.842036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_timeout -r 2 [ 2456.891770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_timeout -r 2 [ 2456.934490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_timeout -r 2 [ 2456.982539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_timeout -r 2 [ 2457.055811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 2457.520915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2457.567536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2457.634697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2457.693751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2457.735688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2457.820685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2457.913976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2457.979333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2458.078719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2458.162346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2458.225583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2458.282342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2458.325262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2458.365828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2459.115885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_dropped -r 2 [ 2459.164481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e remote_dropped -r 2 [ 2459.236587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_dropped -r 2 [ 2459.329458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e remote_dropped -r 2 [ 2459.409932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e remote_dropped -r 2 [ 2459.490652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e remote_dropped -r 2 [ 2459.587467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e remote_dropped -r 2 [ 2459.670643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e remote_dropped -r 2 [ 2459.729584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 2459.776039] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 2459.782813] LNet: Skipped 1 previous similar message [ 2459.797501] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.129@tcp [ 2460.163692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2460.209072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2460.253844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2460.301763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2460.360420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2460.418989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2460.474860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2460.526629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2460.548738] LNet: 88765:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2460.558454] LNet: 88765:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 2460.587516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2460.634973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2460.685572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2460.731769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2460.776782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2460.846550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2469.075818] LNet: 88841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2469.093688] LNetError: 88841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2469.112254] LNet: Removed LNI 192.168.204.29@tcp [ 2469.143820] LNet: Removed LNI 192.168.204.29@tcp1 [ 2470.173652] Key type .llcrypt unregistered [ 2470.180153] Key type ._llcrypt unregistered [ 2497.866146] Key type ._llcrypt registered [ 2497.867715] Key type .llcrypt registered [ 2499.512463] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 02:14:24 (1785392064) [ 2500.128146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2500.159453] alg: No test for adler32 (adler32-zlib) [ 2501.193130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2501.268206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2501.304115] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2502.679197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2502.723831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.29@tcp [ 2502.775072] LNetError: 89823:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.29@tcp to recovery [ 2502.933181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.29@tcp [ 2503.123500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2510.975488] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 02:14:36 (1785392076) [ 2511.114451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2511.153540] LNet: 90407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2511.168660] LNetError: 90407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2511.184253] LNet: Removed LNI 192.168.204.29@tcp [ 2511.200646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2511.263491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2511.289603] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2511.922535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2511.937046] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 2512.714687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.29@tcp [ 2512.727802] LNetError: 90678:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.29@tcp to recovery [ 2513.313574] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 2513.315601] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 2513.995138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.29@tcp [ 2525.422526] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 02:14:50 (1785392090) [ 2527.869203] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg429-server [ 2530.956101] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 02:14:56 (1785392096) [ 2533.098301] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg429-server [ 2535.884094] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 02:15:00 (1785392100) [ 2538.657299] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg429-server [ 2541.951528] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 02:15:06 (1785392106) [ 2542.741827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2542.785581] LNet: 92310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2542.800535] LNetError: 92310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2543.843133] LNet: Removed LNI 192.168.204.29@tcp [ 2543.845643] LNetError: 92310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2543.856050] LNet: Removed LNI 192.168.204.29@tcp1 [ 2543.865773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2543.918836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2543.952892] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2544.875378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2544.981902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2545.052851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2545.161218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2545.226993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2545.295742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2554.206990] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 02:15:19 (1785392119) [ 2554.297460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2554.335286] LNet: 93026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2554.348106] LNetError: 93026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2554.371115] LNet: Removed LNI 192.168.204.29@tcp [ 2554.404810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2554.518127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2554.569265] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2564.544916] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 02:15:30 (1785392130) [ 2564.967832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2565.017890] LNet: 93739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2565.025540] LNetError: 93739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2565.059171] LNet: Removed LNI 192.168.204.29@tcp [ 2565.105488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2565.201771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2565.242476] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2575.754979] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 02:15:41 (1785392141) [ 2577.944539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2580.998682] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 02:15:45 (1785392145) [ 2581.814984] LNet: 94775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2581.840970] LNetError: 94775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2581.861436] LNet: Removed LNI 192.168.204.29@tcp [ 2583.315412] Key type .llcrypt unregistered [ 2583.319977] Key type ._llcrypt unregistered [ 2592.805645] Key type ._llcrypt registered [ 2592.807938] Key type .llcrypt registered [ 2593.064441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2593.082177] alg: No test for adler32 (adler32-zlib) [ 2594.137096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2594.199129] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2603.537629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 2611.766506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.129@tcp [ 2613.368578] LNet: 95194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2613.386451] LNetError: 95194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2613.426651] LNet: Removed LNI 192.168.204.29@tcp [ 2613.822666] Key type .llcrypt unregistered [ 2613.824630] Key type ._llcrypt unregistered [ 2628.800288] Key type ._llcrypt registered [ 2628.807996] Key type .llcrypt registered [ 2630.293646] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2632.211916] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2634.690795] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 02:16:40 (1785392200) [ 2637.371794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2639.604754] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 02:16:44 (1785392204) [ 2642.410961] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg429-server [ 2645.193712] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 02:16:50 (1785392210) [ 2648.895812] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg429-server [ 2651.721259] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 02:16:56 (1785392216) [ 2654.412702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2654.446757] alg: No test for adler32 (adler32-zlib) [ 2655.861575] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 2656.375430] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2658.192705] Key type lgssc registered [ 2660.311541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2688.179092] Key type lgssc unregistered [ 2688.548140] LNet: 98368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2688.559787] LNetError: 98368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2688.576795] LNet: Removed LNI 192.168.204.29@tcp [ 2689.642529] Key type .llcrypt unregistered [ 2689.644238] Key type ._llcrypt unregistered [ 2712.543436] Key type ._llcrypt registered [ 2712.547126] Key type .llcrypt registered [ 2714.479891] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 02:17:59 (1785392279) [ 2715.306335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2715.330698] alg: No test for adler32 (adler32-zlib) [ 2716.340784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2716.413841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2716.452231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2716.492735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2716.570670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2724.936743] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 02:18:10 (1785392290) [ 2725.097302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2725.152553] LNet: 99778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2725.211481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2725.308991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2725.397555] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2727.236242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2727.289933] LNet: 99929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2727.302529] LNetError: 99929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2727.332350] LNet: Removed LNI 192.168.204.29@tcp [ 2727.396431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2727.500956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2727.573723] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2728.345216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2728.359459] LNet: 99957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2728.365229] LNetError: 99957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2728.379027] LNet: Removed LNI 192.168.204.29@tcp [ 2728.403781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2728.485632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2728.528233] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2736.902037] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 02:18:21 (1785392301) [ 2737.123515] LNet: 100545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2737.136598] LNetError: 100545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2737.168870] LNet: Removed LNI 192.168.204.29@tcp [ 2738.090540] Key type .llcrypt unregistered [ 2738.095379] Key type ._llcrypt unregistered [ 2747.837941] Key type ._llcrypt registered [ 2747.840584] Key type .llcrypt registered [ 2748.073060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2748.109844] alg: No test for adler32 (adler32-zlib) [ 2749.025598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2749.166263] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2749.171532] LNet: *** cfs_fail_loc=e003, val=100*** [ 2749.741249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2749.755616] LNet: *** cfs_fail_loc=e003, val=100*** [ 2749.758638] LNet: Skipped 2 previous similar messages [ 2749.762893] Ping reply received with mapping: [ 2749.764329] found metadata reply NI[192.168.204.29@tcp] [ 2749.767073] 1d cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2749.771834] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2749.775550] 5a 5a 5a 5a 5a 5a 5a 5a [ 2749.892850] LNet: 100947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2749.898568] LNetError: 100947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2750.963233] LNet: Removed LNI 192.168.204.29@tcp [ 2752.423543] Key type .llcrypt unregistered [ 2752.427249] Key type ._llcrypt unregistered [ 2767.630640] Key type ._llcrypt registered [ 2767.636241] Key type .llcrypt registered [ 2769.648143] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 02:18:54 (1785392334) [ 2770.660601] Key type .llcrypt unregistered [ 2770.664210] Key type ._llcrypt unregistered [ 2779.695522] Key type ._llcrypt registered [ 2779.699421] Key type .llcrypt registered [ 2779.918101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2779.947290] alg: No test for adler32 (adler32-zlib) [ 2781.150138] LNetError: 101992:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2781.598846] Key type .llcrypt unregistered [ 2781.601447] Key type ._llcrypt unregistered [ 2788.142240] Key type ._llcrypt registered [ 2788.143503] Key type .llcrypt registered [ 2789.724766] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 02:19:15 (1785392355) [ 2790.609443] Key type .llcrypt unregistered [ 2790.611635] Key type ._llcrypt unregistered [ 2800.381881] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2800.394094] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2803.078162] Key type ._llcrypt registered [ 2803.080311] Key type .llcrypt registered [ 2803.247763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2803.260541] alg: No test for adler32 (adler32-zlib) [ 2804.152734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2804.232414] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2815.374150] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 02:19:40 (1785392380) [ 2815.790880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2815.828404] LNet: 103774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2815.840138] LNetError: 103774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2815.873213] LNet: Removed LNI 10.1.2.3@tcp [ 2815.939902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2817.278652] Key type .llcrypt unregistered [ 2817.281798] Key type ._llcrypt unregistered [ 2826.413336] Key type ._llcrypt registered [ 2826.414411] Key type .llcrypt registered [ 2826.829767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2826.871246] alg: No test for adler32 (adler32-zlib) [ 2828.084316] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2828.239746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2828.526899] LNet: 104062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2828.532615] LNetError: 104062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2828.547887] LNet: Removed LNI 192.168.204.29@tcp [ 2829.559449] Key type .llcrypt unregistered [ 2829.571649] Key type ._llcrypt unregistered [ 2849.182287] Key type ._llcrypt registered [ 2849.183343] Key type .llcrypt registered [ 2852.229756] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 02:20:15 (1785392415) [ 2853.793846] Key type .llcrypt unregistered [ 2853.795929] Key type ._llcrypt unregistered [ 2866.379084] Key type ._llcrypt registered [ 2866.382548] Key type .llcrypt registered [ 2866.663770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2866.695684] alg: No test for adler32 (adler32-zlib) [ 2878.198215] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 02:20:42 (1785392442) [ 2878.469520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2878.555322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2878.634915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2878.664386] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2879.541510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2879.596560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2879.657590] LNet: Removed LNI 192.168.204.29@tcp [ 2880.166261] LNet: 105810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2880.967774] Key type .llcrypt unregistered [ 2880.972718] Key type ._llcrypt unregistered [ 2897.323745] Key type ._llcrypt registered [ 2897.326253] Key type .llcrypt registered [ 2899.437957] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 02:21:04 (1785392464) [ 2900.167815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2900.185662] alg: No test for adler32 (adler32-zlib) [ 2901.246695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2901.360699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2901.407205] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2902.347387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2902.421595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.29@tcp [ 2902.441364] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2902.446803] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2902.451222] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2902, 0, 0 [ 2902.458371] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2902.463796] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2902.469501] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2902.471525] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2902, 0, 0 [ 2902.483558] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2902.489705] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2902.498764] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2902.507582] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2902, 0, 2952 [ 2902.518927] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2902.525447] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2902.532678] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2902.539123] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2902, 0, 2952 [ 2902.547361] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2911.275666] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 02:21:17 (1785392477) [ 2912.389042] LNet: 107352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2912.393405] LNetError: 107352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2912.421710] LNet: Removed LNI 192.168.204.29@tcp [ 2913.604449] Key type .llcrypt unregistered [ 2913.606307] Key type ._llcrypt unregistered [ 2922.657315] Key type ._llcrypt registered [ 2922.671045] Key type .llcrypt registered [ 2923.077539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2923.104282] alg: No test for adler32 (adler32-zlib) [ 2924.023791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2924.100994] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2935.235614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 2950.266506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2950.354084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2950.398662] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 2952.006420] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2952.184040] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2952.189770] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2952.200409] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2954.879465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2954.960387] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2980.122214] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 02:22:25 (1785392545) [ 2980.771037] LNet: 108866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2980.785355] LNetError: 108866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2981.793046] LNet: Removed LNI 192.168.204.29@tcp [ 2981.795348] LNetError: 108866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2981.831573] LNet: Removed LNI 192.168.204.29@tcp1 [ 2981.843290] LNet: Removed LNI 10.1.2.3@tcp99 [ 2982.789681] Key type .llcrypt unregistered [ 2982.791416] Key type ._llcrypt unregistered [ 2991.788162] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2991.801699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2994.640880] Key type ._llcrypt registered [ 2994.643956] Key type .llcrypt registered [ 2994.965602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2994.994712] alg: No test for adler32 (adler32-zlib) [ 2996.041645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2996.132094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2996.172173] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2996.932814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2996.949515] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2999.321481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2999.418490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2999.491273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2999.536874] LNetError: 109611:0:(peer.c:2277:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2999.584582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2999.645899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3000.245688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3000.386645] LNet: 109631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3000.398731] LNetError: 109631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3000.409398] LNet: Removed LNI 10.1.2.3@tcp [ 3001.452656] LNet: Removed LNI 192.168.204.29@tcp [ 3002.579494] Key type .llcrypt unregistered [ 3002.585735] Key type ._llcrypt unregistered [ 3022.592490] Key type ._llcrypt registered [ 3022.593843] Key type .llcrypt registered [ 3025.529822] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 02:23:09 (1785392589) [ 3026.981893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3027.025396] alg: No test for adler32 (adler32-zlib) [ 3028.203920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3028.354742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3028.409557] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3029.631679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 3029.775331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg429-client@tcp [ 3041.977585] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 02:23:26 (1785392606) [ 3042.159315] LNet: 111157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3042.183664] LNetError: 111157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3042.232627] LNet: Removed LNI 192.168.204.29@tcp [ 3043.228773] Key type .llcrypt unregistered [ 3043.231546] Key type ._llcrypt unregistered [ 3051.357849] Key type ._llcrypt registered [ 3051.365211] Key type .llcrypt registered [ 3051.658280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3051.685996] alg: No test for adler32 (adler32-zlib) [ 3052.680056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 3052.720485] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3052.750376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 3062.593759] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 02:23:47 (1785392627) [ 3064.705393] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 3067.993850] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 02:23:52 (1785392632) [ 3070.941404] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 3074.881921] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 02:23:58 (1785392638) [ 3075.118917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3075.219355] LNet: 112582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3075.231934] LNetError: 112582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3075.269981] LNet: Removed LNI 192.168.204.29@tcp [ 3075.347402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3075.513855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3075.620041] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3078.139334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 3078.322561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 3095.287878] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 02:24:19 (1785392659) [ 3095.467949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3095.531708] LNet: 113293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3095.553271] LNetError: 113293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3095.617757] LNet: Removed LNI 192.168.204.29@tcp [ 3095.763130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3095.903277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 3095.988323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3096.637858] LNet: 113336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3097.918616] Key type .llcrypt unregistered [ 3097.923157] Key type ._llcrypt unregistered [ 3114.001468] Key type ._llcrypt registered [ 3114.005916] Key type .llcrypt registered [ 3115.519533] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 02:24:41 (1785392681) [ 3116.044711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3116.062244] alg: No test for adler32 (adler32-zlib) [ 3117.162745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3117.269236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3117.319340] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3118.216898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3118.319842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3127.062576] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 02:24:52 (1785392692) [ 3127.327491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3127.362763] LNet: 114856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3127.379370] LNetError: 114856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3128.432666] LNet: Removed LNI 192.168.204.29@tcp [ 3128.496260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3128.598577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 3128.653717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 3136.960981] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 02:25:02 (1785392702) [ 3137.173676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3137.231206] LNet: 115430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3137.324666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3137.535349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3138.687178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3138.792744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3138.999513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 3140.090397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3140.191723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3140.395120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3140.491036] LNet: 115470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3140.549466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3140.693685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3141.702619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3141.767583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3141.875197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 3142.990557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3143.070162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3154.218101] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 02:25:19 (1785392719) [ 3154.695482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3154.739181] LNet: 116071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3154.788355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3154.917924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3154.994977] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3156.812366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3156.872805] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 3158.173264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3158.263847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3158.630642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3158.671753] LNetError: 116351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3158.687771] LNet: Removed LNI 192.168.204.29@tcp [ 3158.696276] LNet: Removed LNI 192.168.204.29@tcp2 [ 3158.733855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3158.797135] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3158.802765] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 3158.806711] LNet: 116357:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3159.253885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3159.500611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3159.531164] LNet: 116394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3159.539509] LNet: 116394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 3160.610262] LNet: Removed LNI 192.168.204.29@tcp [ 3160.616580] LNetError: 116394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3160.633552] LNetError: 116394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 3160.674474] LNet: Removed LNI 192.168.204.29@tcp2 [ 3160.727068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3160.868322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3160.905906] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3161.667798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3161.684955] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 3162.607634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3162.693407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 3162.782904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3163.037826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3163.061037] LNetError: 116678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3163.068749] LNet: Removed LNI 192.168.204.29@tcp [ 3164.146734] LNet: Removed LNI 192.168.204.29@tcp2 [ 3164.224353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3164.386545] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3164.404786] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 3164.411369] LNet: 116685:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3165.566503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3165.813146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3165.840681] LNet: Removed LNI 192.168.204.29@tcp [ 3166.894159] LNet: Removed LNI 192.168.204.29@tcp2 [ 3166.943361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3167.058651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3167.103791] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3168.428919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3168.465874] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 3169.928482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3170.012872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3170.410628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3170.461562] LNet: 117001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3170.477245] LNet: 117001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 3170.491260] LNetError: 117001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3170.493848] LNetError: 117001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 3170.504833] LNet: Removed LNI 192.168.204.29@tcp [ 3170.522383] LNet: Removed LNI 192.168.204.29@tcp2 [ 3170.583974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3170.694757] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3170.702602] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 3170.710706] LNet: 117007:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3171.894965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3172.308694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3173.408749] LNet: Removed LNI 192.168.204.29@tcp [ 3173.427843] LNet: Removed LNI 192.168.204.29@tcp2 [ 3173.502796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3173.626053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3173.681989] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3175.011625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3175.035727] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 3176.557608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3176.638436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 3176.755991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3177.254985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3178.338191] LNet: Removed LNI 192.168.204.29@tcp [ 3179.375650] LNet: Removed LNI 192.168.204.29@tcp2 [ 3179.442863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3179.589370] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3179.594721] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 3179.605882] LNet: 117336:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3180.511496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3180.942650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3180.975773] LNetError: 117373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3180.982215] LNetError: 117373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 3180.991549] LNet: Removed LNI 192.168.204.29@tcp [ 3182.057303] LNet: Removed LNI 192.168.204.29@tcp2 [ 3182.154352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3182.269520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3182.316454] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3183.745715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3183.814148] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 3185.714847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3185.764703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3185.891517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3186.702497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3186.774407] LNet: 117670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3186.783534] LNet: 117670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 3187.808260] LNet: Removed LNI 192.168.204.29@tcp [ 3187.835863] LNet: Removed LNI 192.168.204.29@tcp2 [ 3187.889566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3187.981352] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3187.995241] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 3188.003131] LNet: 117676:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3188.790609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3189.442221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3190.497953] LNet: Removed LNI 192.168.204.29@tcp [ 3190.517787] LNet: Removed LNI 192.168.204.29@tcp2 [ 3190.617674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3190.770275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3190.848785] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3191.951573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3191.991977] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 3193.441586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3193.530240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 3193.617556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 3193.739940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3194.364656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3195.424460] LNet: Removed LNI 192.168.204.29@tcp [ 3196.456476] LNet: Removed LNI 192.168.204.29@tcp2 [ 3196.527169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3196.667517] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3196.676499] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 3196.685914] LNet: 118036:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3197.904682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3209.786799] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 3212.416080] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 02:26:17 (1785392777) [ 3212.711887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3212.752226] LNetError: 118795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3212.763101] LNetError: 118795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 3212.781561] LNet: Removed LNI 192.168.204.29@tcp [ 3213.869930] LNet: Removed LNI 192.168.204.29@tcp2 [ 3213.918987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3214.372832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3214.443547] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3215.878985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3215.922489] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3217.132300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3217.172041] LNet: Removed LNI 10.1.2.3@tcp [ 3217.192326] LNet: Removed LNI 192.168.204.29@tcp [ 3217.246193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3217.339898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 3217.418616] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3217.434500] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3218.480468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 3218.514256] LNet: Removed LNI 192.168.204.29@tcp [ 3218.533456] LNet: Removed LNI 10.1.2.3@tcp [ 3226.571313] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 02:26:32 (1785392792) [ 3226.870656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3226.889498] LNet: 119673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3226.893650] LNet: 119673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 3226.908941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3226.955872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3226.984218] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3227.682946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 3227.717150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 3227.749793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 3227.783307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 3227.815637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 3227.852511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 3227.899163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 3227.935548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 3227.978926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 3228.042398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 3228.079904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 3228.293526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 3237.347300] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 02:26:42 (1785392802) [ 3237.606975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3238.695790] LNet: Removed LNI 192.168.204.29@tcp [ 3238.822315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3239.049733] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 3239.588723] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3239.603459] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3239.620124] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 3242.349410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3242.402258] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3243.958937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3243.977876] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3247.325632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 3247.389592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 3247.551598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.29@tcp -m PUT -e local_timeout -r 1 [ 3247.679687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 3247.839364] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3247.851131] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3248.208803] LNetError: 120478:0:(lib-move.c:4801:lnet_parse()) 192.168.204.29@tcp, src 192.168.204.29@tcp: Dropping PUT (error -108 looking up sender) [ 3249.251050] LNetError: 120936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3249.258807] LNetError: 120936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 3249.269427] LNet: Removed LNI 10.1.2.3@tcp [ 3250.341650] LNet: Removed LNI 192.168.204.29@tcp [ 3250.953147] Key type .llcrypt unregistered [ 3250.960229] Key type ._llcrypt unregistered [ 3273.078849] Key type ._llcrypt registered [ 3273.080888] Key type .llcrypt registered [ 3275.964733] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 02:27:19 (1785392839) [ 3277.034667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3277.094064] alg: No test for adler32 (adler32-zlib) [ 3278.589207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3279.484305] LNet: 121792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3280.584457] Key type .llcrypt unregistered [ 3280.585813] Key type ._llcrypt unregistered [ 3292.173668] Key type ._llcrypt registered [ 3292.177706] Key type .llcrypt registered [ 3292.657508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3292.717275] alg: No test for adler32 (adler32-zlib) [ 3293.847231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3293.907610] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3307.456052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 3307.755771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3309.367823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.129@tcp [ 3310.095394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3310.161787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3310.275704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3310.320057] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.129@tcp [ 3310.415856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3310.477858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3310.561590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3310.642408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3310.778199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3310.830660] LNet: 122245:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3310.891108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3310.954250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3310.993037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3311.071388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3311.114385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3311.169349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3311.954760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3312.215295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3312.247278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3312.271703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3312.310415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3312.333918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3312.359608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3312.406318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3312.442720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3312.461178] LNet: 122363:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3312.478704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3312.525317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3312.586306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3312.634753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3312.667246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3312.704284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3313.755073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.129@tcp [ 3314.301759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3314.367983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3314.429711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3314.512751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3314.583963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3314.674433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3314.813181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3314.833564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3314.858716] LNet: 122480:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3314.886065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3314.932246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3314.984306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3315.046456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3315.130436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3315.238266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3316.656952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3317.273443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3317.378779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3317.470195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3317.558707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3317.621895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3317.705125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3317.775905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3317.877665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3317.909428] LNet: 122598:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3317.949599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3318.044908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3318.127290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3318.219930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3318.300503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3318.450834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3318.996870] LNet: 122644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3319.012082] LNetError: 122644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3319.042617] LNet: Removed LNI 192.168.204.29@tcp [ 3320.025810] Key type .llcrypt unregistered [ 3320.032261] Key type ._llcrypt unregistered [ 3346.788364] Key type ._llcrypt registered [ 3346.794214] Key type .llcrypt registered [ 3348.393773] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 02:28:33 (1785392913) [ 3349.038466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3349.055065] alg: No test for adler32 (adler32-zlib) [ 3350.044811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3350.728040] LNet: 123534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3351.501081] Key type .llcrypt unregistered [ 3351.503728] Key type ._llcrypt unregistered [ 3359.235621] Key type ._llcrypt registered [ 3359.238626] Key type .llcrypt registered [ 3359.580202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3359.641846] alg: No test for adler32 (adler32-zlib) [ 3360.712915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3360.754565] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3372.510432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 3389.773890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3389.805668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3389.853747] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 3391.436538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3392.383022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.129@tcp [ 3392.545718] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.129@tcp [ 3392.808375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3392.869218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3392.907956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3392.938663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3392.993336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3393.036373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3393.067047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3393.083270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3393.095450] LNet: 124189:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3393.107302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3393.132243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3393.157404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3393.175910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3393.219606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3393.261779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3393.803700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3394.286828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3394.349467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3394.410684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3394.463763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3394.504346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3394.546867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3394.586021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3394.667765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3394.684942] LNet: 124307:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3394.699204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3394.729878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3394.767407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3394.798407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3394.852455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3394.906353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3395.782928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.129@tcp [ 3395.978565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3396.004814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3396.052304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3396.075427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3396.098183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3396.168905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3396.204940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3396.243432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3396.256847] LNet: 124424:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3396.283715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3396.334161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3396.379710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3396.409978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3396.435478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3396.469750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3397.418482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3397.678507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3397.706405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3397.741974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3397.777217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3397.821436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3397.918728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3398.031327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3398.127257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3398.209410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3398.282351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3398.380586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3398.468671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3398.545318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3398.619540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3406.265355] LNet: 124622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3406.277847] LNetError: 124622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3406.283836] LNet: Removed LNI 192.168.204.29@tcp [ 3406.293704] LNet: Removed LNI 192.168.204.29@tcp1 [ 3406.849647] Key type .llcrypt unregistered [ 3406.852624] Key type ._llcrypt unregistered [ 3432.699850] Key type ._llcrypt registered [ 3432.704515] Key type .llcrypt registered [ 3434.124534] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 02:29:59 (1785392999) [ 3436.870710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 3440.596137] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 02:30:04 (1785393004) [ 3449.203860] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 02:30:14 (1785393014) [ 3459.098861] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 02:30:24 (1785393024) [ 3467.379861] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 02:30:32 (1785393032) [ 3476.702494] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 02:30:42 (1785393042) [ 3485.493132] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 02:30:50 (1785393050) [ 3495.167420] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 02:31:00 (1785393060) [ 3504.566974] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 02:31:10 (1785393070) [ 3514.490387] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 02:31:19 (1785393079) [ 3524.225927] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 02:31:29 (1785393089) [ 3534.143768] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 02:31:39 (1785393099) [ 3545.127648] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 02:31:50 (1785393110) [ 3554.851534] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 02:32:00 (1785393120) [ 3564.574946] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 02:32:09 (1785393129) [ 3575.708939] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 02:32:20 (1785393140) [ 3584.766861] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 02:32:30 (1785393150) [ 3593.497397] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 02:32:38 (1785393158) [ 3604.207504] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 02:32:48 (1785393168) [ 3604.831490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3604.879483] alg: No test for adler32 (adler32-zlib) [ 3606.021557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3606.146328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3606.171713] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3607.401227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3607.431526] LNet: 135637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3607.453856] LNetError: 135637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3607.474767] LNet: Removed LNI 192.168.204.29@tcp [ 3607.505068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3607.616307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 3607.663806] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3617.817900] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 02:33:02 (1785393182) [ 3627.464072] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 02:33:13 (1785393193) [ 3640.829829] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 02:33:25 (1785393205) [ 3655.035053] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 02:33:39 (1785393219) [ 3666.390695] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 02:33:51 (1785393231) [ 3676.718673] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 02:34:02 (1785393242) [ 3687.530985] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 02:34:13 (1785393253) [ 3696.084722] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 02:34:21 (1785393261) [ 3706.625582] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 02:34:32 (1785393272) [ 3716.950788] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 02:34:41 (1785393281) [ 3727.763769] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 02:34:53 (1785393293) [ 3737.149236] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 02:35:02 (1785393302) [ 3747.942805] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 3395 sec ========== 02:35:12 (1785393312) [ 3748.552267] LNet: 143112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3748.565235] LNetError: 143112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3748.596080] LNet: Removed LNI 192.168.204.29@tcp [ 3749.404145] Key type .llcrypt unregistered [ 3749.405383] Key type ._llcrypt unregistered [ 3769.120146] Key type ._llcrypt registered [ 3769.132265] Key type .llcrypt registered [ 3769.761387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3769.797568] alg: No test for adler32 (adler32-zlib) [ 3771.581159] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 3772.113788] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 3773.944154] Key type lgssc registered [ 3775.845967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3872.030651] Lustre: Mounted lustre-client [ 3879.799887] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3892.840783] Lustre: Unmounted lustre-client [ 3918.906960] Key type lgssc unregistered [ 3919.186609] LNet: 146049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3919.205684] LNetError: 146049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3919.221403] LNet: Removed LNI 192.168.204.29@tcp [ 3920.072134] Key type .llcrypt unregistered [ 3920.074078] Key type ._llcrypt unregistered