[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489197083 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002308] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007017] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.009118] LSM: Security Framework initializing [ 0.010032] Yama: becoming mindful. [ 0.011030] SELinux: Initializing. [ 0.012059] *** VALIDATE selinux *** [ 0.019081] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024392] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025134] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026089] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027096] *** VALIDATE tmpfs *** [ 0.029148] *** VALIDATE proc *** [ 0.030130] *** VALIDATE cgroup *** [ 0.031006] *** VALIDATE cgroup2 *** [ 0.032228] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033129] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036022] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.040019] debug: unmapping init [mem 0xffffffff8aa59000-0xffffffff8aa60fff] [ 0.042200] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043573] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044018] ... version: 2 [ 0.045010] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.047009] ... value mask: 0000ffffffffffff [ 0.048008] ... max period: 00007fffffffffff [ 0.049008] ... fixed-purpose events: 3 [ 0.050008] ... event mask: 000000070000000f [ 0.051262] rcu: Hierarchical SRCU implementation. [ 0.053255] smp: Bringing up secondary CPUs ... [ 0.054493] x86: Booting SMP configuration: [ 0.055024] .... node #0, CPUs: #1 #2 #3 [ 0.061013] smp: Brought up 1 node, 4 CPUs [ 0.063009] smpboot: Max logical packages: 1 [ 0.064008] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.146019] node 0 deferred pages initialised in 79ms [ 0.149095] devtmpfs: initialized [ 0.150226] x86/mm: Memory block size: 128MB [ 0.153502] gcov: version magic: 0x41383552 [ 0.157276] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158065] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159249] pinctrl core: initialized pinctrl subsystem [ 0.160141] [ 0.160644] ************************************************************* [ 0.161008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162009] ** ** [ 0.163007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164008] ** ** [ 0.165008] ** This means that this kernel is built to expose internal ** [ 0.166010] ** IOMMU data structures, which may compromise security on ** [ 0.167009] ** your system. ** [ 0.168009] ** ** [ 0.169007] ** If you see this message and you are not debugging the ** [ 0.170008] ** kernel, report this immediately to your vendor! ** [ 0.171007] ** ** [ 0.172008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173008] ************************************************************* [ 0.174587] NET: Registered protocol family 16 [ 0.175376] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176037] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.177041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179032] cpuidle: using governor menu [ 0.180535] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182563] PCI: Using configuration type 1 for base access [ 0.186113] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196364] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.199017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.202141] cryptd: max_cpu_qlen set to 1000 [ 0.206341] ACPI: Added _OSI(Module Device) [ 0.207011] ACPI: Added _OSI(Processor Device) [ 0.209010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.211010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.217605] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.223352] ACPI: Interpreter enabled [ 0.224046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.226011] ACPI: Using IOAPIC for interrupt routing [ 0.228080] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.231518] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.242542] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.245032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.249017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.252070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258080] acpiphp: Slot [2] registered [ 0.259101] acpiphp: Slot [5] registered [ 0.261154] acpiphp: Slot [6] registered [ 0.263178] acpiphp: Slot [3] registered [ 0.265146] acpiphp: Slot [4] registered [ 0.268156] acpiphp: Slot [7] registered [ 0.269105] acpiphp: Slot [8] registered [ 0.271093] acpiphp: Slot [9] registered [ 0.273093] acpiphp: Slot [10] registered [ 0.275075] acpiphp: Slot [11] registered [ 0.277060] acpiphp: Slot [12] registered [ 0.279204] acpiphp: Slot [13] registered [ 0.281092] acpiphp: Slot [14] registered [ 0.283100] acpiphp: Slot [15] registered [ 0.285123] acpiphp: Slot [16] registered [ 0.287129] acpiphp: Slot [17] registered [ 0.289097] acpiphp: Slot [18] registered [ 0.291093] acpiphp: Slot [19] registered [ 0.293121] acpiphp: Slot [20] registered [ 0.295164] acpiphp: Slot [21] registered [ 0.297114] acpiphp: Slot [22] registered [ 0.299097] acpiphp: Slot [23] registered [ 0.301113] acpiphp: Slot [24] registered [ 0.303084] acpiphp: Slot [25] registered [ 0.305072] acpiphp: Slot [26] registered [ 0.308086] acpiphp: Slot [27] registered [ 0.310146] acpiphp: Slot [28] registered [ 0.312122] acpiphp: Slot [29] registered [ 0.314117] acpiphp: Slot [30] registered [ 0.316119] acpiphp: Slot [31] registered [ 0.318103] PCI host bridge to bus 0000:00 [ 0.320016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.323017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.327017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.330017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333016] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.337023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.340177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.344000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.349138] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.357012] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.362117] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.367019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.369011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.376496] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.378702] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382030] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.384636] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.388871] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.397909] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.405015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.412550] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.419013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425010] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.440013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.453575] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.462011] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.470017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.498013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.514000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.516289] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.518263] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.520268] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.522171] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.526167] iommu: Default domain type: Passthrough [ 0.528412] SCSI subsystem initialized [ 0.530118] ACPI: bus type USB registered [ 0.532074] usbcore: registered new interface driver usbfs [ 0.533057] usbcore: registered new interface driver hub [ 0.535053] usbcore: registered new device driver usb [ 0.536138] pps_core: LinuxPPS API ver. 1 registered [ 0.537007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.540061] PTP clock support registered [ 0.543055] EDAC MC: Ver: 3.0.0 [ 0.545130] PCI: Using ACPI for IRQ routing [ 0.546619] NetLabel: Initializing [ 0.548009] NetLabel: domain hash size = 128 [ 0.550007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.552065] NetLabel: unlabeled traffic allowed by default [ 0.555123] vgaarb: loaded [ 0.557253] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.558008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.563318] clocksource: Switched to clocksource kvm-clock [ 0.659782] VFS: Disk quotas dquot_6.6.0 [ 0.661905] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.665929] *** VALIDATE ramfs *** [ 0.667772] *** VALIDATE hugetlbfs *** [ 0.669929] pnp: PnP ACPI init [ 0.672757] pnp: PnP ACPI: found 6 devices [ 0.691050] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.695567] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.698030] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.700650] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.703499] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.706145] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.709085] NET: Registered protocol family 2 [ 0.711780] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.715467] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.719418] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.726224] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.730726] TCP: Hash tables configured (established 65536 bind 65536) [ 0.733715] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.736325] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.739228] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.742192] NET: Registered protocol family 1 [ 0.745541] RPC: Registered named UNIX socket transport module. [ 0.748716] RPC: Registered udp transport module. [ 0.751603] RPC: Registered tcp transport module. [ 0.754197] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.757942] NET: Registered protocol family 44 [ 0.760673] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.763938] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.766966] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.769966] PCI: CLS 0 bytes, default 64 [ 0.772587] Unpacking initramfs... [ 2.172222] debug: unmapping init [mem 0xffff92533cc64000-0xffff92533ffcffff] [ 2.176251] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.179035] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.181346] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.681157] Initialise system trusted keyrings [ 2.683079] Key type blacklist registered [ 2.684782] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.693510] zbud: loaded [ 2.695892] *** VALIDATE nfs *** [ 2.696815] *** VALIDATE nfs4 *** [ 2.698167] pstore: using deflate compression [ 2.701549] Platform Keyring initialized [ 2.798911] NET: Registered protocol family 38 [ 2.801227] Key type asymmetric registered [ 2.803267] Asymmetric key parser 'x509' registered [ 2.805692] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.809743] io scheduler mq-deadline registered [ 2.811764] io scheduler kyber registered [ 2.813919] io scheduler bfq registered [ 2.816396] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.820173] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.822881] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.825641] ACPI: Power Button [PWRF] [ 2.830952] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.836329] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.893366] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.925218] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.952283] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.959945] Non-volatile memory driver v1.3 [ 2.962077] Linux agpgart interface v0.103 [ 2.992645] virtio_blk virtio1: [vda] 146008 512-byte logical blocks (74.8 MB/71.3 MiB) [ 2.994948] vda: detected capacity change from 0 to 74756096 [ 3.008251] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.011530] vdb: detected capacity change from 0 to 1073741824 [ 3.018331] libphy: Fixed MDIO Bus: probed [ 3.023106] usbcore: registered new interface driver usbserial_generic [ 3.025371] usbserial: USB Serial support registered for generic [ 3.027512] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.032607] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.034228] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.036712] mousedev: PS/2 mouse device common for all mice [ 3.039622] rtc_cmos 00:05: RTC can wake from S4 [ 3.043487] rtc_cmos 00:05: registered as rtc0 [ 3.046228] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.050214] intel_pstate: CPU model not supported [ 3.052823] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.056152] hid: raw HID events driver (C) Jiri Kosina [ 3.062362] usbcore: registered new interface driver usbhid [ 3.063858] usbhid: USB HID core driver [ 3.065122] drop_monitor: Initializing network drop monitor service [ 3.067979] Initializing XFRM netlink socket [ 3.071011] NET: Registered protocol family 10 [ 3.072688] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.074743] Segment Routing with IPv6 [ 3.078175] NET: Registered protocol family 17 [ 3.080785] mpls_gso: MPLS GSO support [ 3.083821] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.088900] RAS: Correctable Errors collector initialized. [ 3.091977] AVX version of gcm_enc/dec engaged. [ 3.094689] AES CTR mode by8 optimization enabled [ 3.190759] sched_clock: Marking stable (3190739930, 0)->(4045840240, -855100310) [ 3.194752] registered taskstats version 1 [ 3.197294] Loading compiled-in X.509 certificates [ 3.199378] zswap: loaded using pool lzo/zbud [ 3.227138] Key type big_key registered [ 3.240073] Key type encrypted registered [ 3.241213] ima: No TPM chip found, activating TPM-bypass! [ 3.242793] ima: Allocated hash algorithm: sha1 [ 3.243986] ima: No architecture policies found [ 3.245083] evm: Initialising EVM extended attributes: [ 3.246426] evm: security.selinux [ 3.247167] evm: security.ima [ 3.247881] evm: security.capability [ 3.248809] evm: HMAC attrs: 0x1 [ 3.252140] rtc_cmos 00:05: setting system clock to 2026-08-06 16:35:00 UTC (1786034100) [ 3.257040] debug: unmapping init [mem 0xffffffff8ba03000-0xffffffff8bbfffff] [ 3.259307] debug: unmapping init [mem 0xffffffff8a782000-0xffffffff8aa58fff] [ 3.269344] Write protecting the kernel read-only data: 28672k [ 3.272321] debug: unmapping init [mem 0xffffffff88e03000-0xffffffff88ffffff] [ 3.274323] debug: unmapping init [mem 0xffffffff89714000-0xffffffff897fffff] [ 3.302721] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.309018] systemd[1]: Detected virtualization kvm. [ 3.310819] systemd[1]: Detected architecture x86-64. [ 3.312270] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.340598] systemd[1]: No hostname configured. [ 3.342041] systemd[1]: Set hostname to . [ 3.343572] random: systemd: uninitialized urandom read (16 bytes read) [ 3.345260] systemd[1]: Initializing machine ID from random generator. [ 3.377910] random: ln: uninitialized urandom read (6 bytes read) [ 3.463810] random: systemd: uninitialized urandom read (16 bytes read) [ 3.467970] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.475700] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.485026] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.107748] device-mapper: uevent: version 1.0.3 [ 4.110388] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.721463] virtio_net virtio0 ens2: renamed from eth0 [ 4.729358] scsi host0: ata_piix [ 4.763801] scsi host1: ata_piix [ 4.766606] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.769782] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.977254] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.757541] random: crng init done [ 9.759051] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.031800] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.124140] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.418713] SELinux: Disabled at runtime. [ 11.494252] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.502794] systemd[1]: Detected virtualization kvm. [ 11.504678] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.210139] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.213499] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.218875] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.222610] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.227322] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.238981] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.245441] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ 12.372904] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.680041] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.986316] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.024862] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.101299] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.120434] EDAC sbridge: Ver: 1.1.2 [ 14.157199] Key type dns_resolver registered [ 14.467246] NFS: Registering the id_resolver key type [ 14.470138] Key type id_resolver registered [ 14.472370] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg442-client login: [ 40.802554] libcfs: loading out-of-tree module taints kernel. [ 40.845850] Key type ._llcrypt registered [ 40.847469] Key type .llcrypt registered [ 41.036676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 41.044341] alg: No test for adler32 (adler32-zlib) [ 42.095992] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 42.456293] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 44.087172] Key type lgssc registered [ 44.837946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.699072] hrtimer: interrupt took 5995348 ns [ 206.910864] Lustre: Mounted lustre-client - version 2.17.56_53_g253671b [ 214.170226] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 231.869497] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing check_logdir /tmp/testlogs/ [ 232.415259] Lustre: lustre-OST0000-osc-ffff925385491000: disconnect after 22s idle [ 239.128517] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing yml_node [ 245.397857] Lustre: DEBUG MARKER: Client: 2.17.56.53 [ 247.883283] Lustre: DEBUG MARKER: MDS: 2.17.56.53 [ 250.815444] Lustre: DEBUG MARKER: OSS: 2.17.56.53 [ 253.038712] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Aug 6 12:39:08 EDT 2026 [ 273.320974] Lustre: Unmounted lustre-client [ 300.112979] Key type lgssc unregistered [ 300.443838] LNet: 8421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.452940] LNetError: 8421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 300.480640] LNet: Removed LNI 192.168.204.42@tcp [ 301.711140] Key type .llcrypt unregistered [ 301.714815] Key type ._llcrypt unregistered [ 339.564944] Key type ._llcrypt registered [ 339.569467] Key type .llcrypt registered [ 339.943398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 339.998811] alg: No test for adler32 (adler32-zlib) [ 341.062763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 341.215103] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 341.275188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 347.397407] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 347.624859] LNet: 9323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.633899] LNetError: 9323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 347.705353] LNet: Removed LNI 192.168.204.42@tcp [ 348.745822] Key type .llcrypt unregistered [ 348.747764] Key type ._llcrypt unregistered [ 358.914361] Key type ._llcrypt registered [ 358.921432] Key type .llcrypt registered [ 360.903579] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:40:55 (1786034455) [ 361.365681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 361.410405] alg: No test for adler32 (adler32-zlib) [ 362.633514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.907159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.001163] LNet: 9744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.063984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.247615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 363.351776] LNet: 9751:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 373.641284] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:41:08 (1786034468) [ 373.778193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.821666] LNet: 10322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.880183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.961940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 384.399908] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:41:19 (1786034479) [ 384.939782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.983587] LNet: 10918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.999908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.162989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 397.788595] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:41:31 (1786034491) [ 398.294418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.443521] LNet: 11500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.536277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.962091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 413.638796] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:41:48 (1786034508) [ 414.071258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.119072] LNet: 12089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.176130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.331123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 415.314578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 425.750425] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:42:01 (1786034521) [ 426.150763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.220151] LNet: 12680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.256523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.368380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 427.609808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 428.473317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 441.570478] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:42:15 (1786034535) [ 441.725837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.837862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.126225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 455.491151] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:42:30 (1786034550) [ 455.842675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.955675] LNet: 13864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 455.962577] LNet: 13864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 456.021740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.142255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 456.183837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 456.876710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 456.920873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 457.652533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 457.714528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 458.236695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 458.293783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 458.875889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 458.963073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 459.474414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 459.509100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 459.911265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 459.956865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 460.525360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 460.572180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 460.937635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 461.068161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 461.996459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 462.073437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 462.847849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 462.927674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 463.587734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 463.639041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 464.401978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 464.553317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 476.864793] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:42:51 (1786034571) [ 476.962768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.036784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.225994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 477.331457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 486.880630] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:43:02 (1786034582) [ 487.269437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.320382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.415719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 487.466833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 496.736843] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:43:11 (1786034591) [ 496.848157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.879672] LNet: 15714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 496.892302] LNet: 15714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 496.949913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.092785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 497.162115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 506.316648] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:43:21 (1786034601) [ 506.586959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.698520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.957613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 507.027205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 516.435127] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:43:31 (1786034611) [ 516.877585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.960866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.136668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 517.196443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 528.553526] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:43:43 (1786034623) [ 528.714157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.818964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.965882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 529.026738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 539.249251] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:43:54 (1786034634) [ 539.771307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.819566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.943239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 540.500193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 550.143685] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:44:05 (1786034645) [ 550.440328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.560499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.970289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 551.470699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 561.501699] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:44:16 (1786034656) [ 561.673252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.727639] LNet: 19281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 561.742723] LNet: 19281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 561.796951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.036630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 562.369556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 571.887553] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:44:27 (1786034667) [ 572.047523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.097166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.631676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 572.926304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 584.904277] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:44:40 (1786034680) [ 585.200324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.234528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.344542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 585.414465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 594.731986] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:44:50 (1786034690) [ 595.262817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.380571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.530358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 595.623897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 604.115781] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:44:59 (1786034699) [ 604.209081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.254780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.367062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 613.624887] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:45:08 (1786034708) [ 613.920408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.027427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.290532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 623.199963] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:45:18 (1786034718) [ 623.350331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.417689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.568410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 624.205770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 633.473982] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:45:29 (1786034729) [ 634.081922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.114737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.188235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 634.255577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 643.692552] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:45:38 (1786034738) [ 643.821177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.919757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.057488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 644.114740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 653.801142] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:45:49 (1786034749) [ 653.947595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.011103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.196271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 654.279180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 664.125098] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:45:59 (1786034759) [ 664.266983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.326788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.475694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 664.549648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 671.974622] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:46:07 (1786034767) [ 672.137647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.204689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.321375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 672.386721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 672.408471] LNetError: 25876:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 672.448697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 672.481892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 682.064562] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:46:16 (1786034776) [ 682.300940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.374759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.476570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.EZ9865UwNC [ 682.646196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.EZ9865UwNC [ 682.751669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.EZ9865UwNC [ 691.284906] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:46:26 (1786034786) [ 691.522016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.558846] LNet: 27047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.566464] LNet: 27047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 691.598681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.714287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 691.779596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 691.832805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 699.554073] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 12:46:35 (1786034795) [ 700.146960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.232508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.371857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 700.413762] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 702.574136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 704.264946] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 706.407218] Key type lgssc registered [ 707.745753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 711.466672] LNetError: 27639:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.42@tcp99 has no available nets [ 714.799551] Key type lgssc unregistered [ 715.370816] LNetError: 28750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 716.394242] LNet: Removed LNI 192.168.204.42@tcp [ 717.388622] Key type .llcrypt unregistered [ 717.391749] Key type ._llcrypt unregistered [ 726.515596] Key type ._llcrypt registered [ 726.517814] Key type .llcrypt registered [ 728.184116] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:47:03 (1786034823) [ 728.648842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 728.668822] alg: No test for adler32 (adler32-zlib) [ 729.631446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.720379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 734.094218] LNet: Message forwarding will be enabled in 2s [ 737.183170] LNet: Message forwarding enabled [ 737.841227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 737.907102] LNet: Message forwarding will stop in 2s to 6s [ 741.601682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 744.278673] LNet: Message forwarding disabled [ 745.806585] LNet: Message forwarding will be enabled in 2s [ 749.413967] LNet: Message forwarding enabled [ 758.214812] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:47:32 (1786034852) [ 758.406988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.467535] LNet: 30145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.678613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.925791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 772.047633] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:47:47 (1786034867) [ 772.194685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.245475] LNet: 30728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.314553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.432200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 772.497665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 772.563509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 772.633443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 772.742718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 772.805636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 772.874074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 772.941861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 773.026849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 773.098816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 773.159849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 773.250887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 773.333502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 773.417648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 773.523119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 773.656624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 773.765195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 773.837813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 773.944717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 773.993619] LNetError: 30777:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 783.198701] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:47:58 (1786034878) [ 783.439406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.508469] LNet: 31346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.558935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.724481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 792.065053] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:48:07 (1786034887) [ 794.838976] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg442-server [ 797.368501] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:48:12 (1786034892) [ 797.688869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.731750] LNet: 32280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.759396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.844444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.892057] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 799.221269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[43-47]@tcp [ 799.279442] LNet: 32413:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 799.333243] LNetError: 32289:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.45@tcp has gone from up to down [ 799.378847] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.46@tcp is being used as a gateway but routing feature is not turned on [ 800.365746] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.46@tcp is being used as a gateway but routing feature is not turned on [ 800.397639] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 801.382345] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.47@tcp is being used as a gateway but routing feature is not turned on [ 801.393303] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 803.444536] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.46@tcp is being used as a gateway but routing feature is not turned on [ 803.469521] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 807.550798] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.45@tcp is being used as a gateway but routing feature is not turned on [ 807.576267] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 808.051544] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:48:23 (1786034903) [ 810.836226] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg442-server [ 812.455910] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:48:28 (1786034908) [ 812.936671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.729877] LNetError: 32283:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.204.45@tcp [ 849.568949] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 852.959298] LNetError: 32290:0:(socklnd_cb.c:1675:ksocknal_recv_hello()) Error -11 reading HELLO from 192.168.204.43 [ 852.959859] LNetError: Unexpected error -11 connecting to 192.168.204.44@tcp at host 192.168.204.44:988 [ 852.980382] LNetError: 32290:0:(socklnd_cb.c:1675:ksocknal_recv_hello()) Skipped 1 previous similar message [ 853.010358] LNetError: Skipped 1 previous similar message [ 853.923687] LNetError: 32284:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.44@tcp has gone from up to down [ 853.939566] LNetError: 32284:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 853.979933] LNet: 33349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.990909] LNetError: 33349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.025105] LNet: Removed LNI 192.168.204.42@tcp [ 854.095305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.201460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 854.247832] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 855.399919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[43-48/2]@tcp [ 855.447919] LNet: 33473:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 855.458101] LNet: 33473:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 855.502950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[43-48/2]@tcp [ 855.510691] LNetError: 33360:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.45@tcp is being used as a gateway but routing feature is not turned on [ 855.538650] LNetError: 33360:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 17 previous similar messages [ 855.550712] LNetError: 33360:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.45@tcp has gone from up to down [ 855.563884] LNetError: 33360:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 866.639433] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:49:21 (1786034961) [ 866.791190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 910.303200] LNetError: 33365:0:(socklnd_cb.c:1675:ksocknal_recv_hello()) Error -11 reading HELLO from 192.168.204.43 [ 910.323232] LNetError: Unexpected error -11 connecting to 192.168.204.43@tcp at host 192.168.204.43:988 [ 910.815445] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 910.880174] LNet: 34058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.899508] LNetError: 34058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.920423] LNet: Removed LNI 192.168.204.42@tcp [ 910.995759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.081193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 911.159730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.236553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.356159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 911.421215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.464877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.558159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 911.679292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.765682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.858717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 911.964920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.018904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.089639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 912.223460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.308084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.462231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 912.613748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.673622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.744509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 912.839604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.884484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.994388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 913.082965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.142769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.282396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 913.441209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.514868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.667821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 913.828570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.904083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.038282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 914.169288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.295394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.438619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 925.058363] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:50:20 (1786035020) [ 927.447523] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg442-server [ 930.214185] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:50:25 (1786035025) [ 932.120009] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg442-server [ 934.613706] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:50:29 (1786035029) [ 934.788196] LNet: 35468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.795201] LNet: 35468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 935.925721] Key type .llcrypt unregistered [ 935.932056] Key type ._llcrypt unregistered [ 945.627011] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 945.917878] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 945.935494] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 945.953909] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 949.139945] Key type ._llcrypt registered [ 949.144670] Key type .llcrypt registered [ 949.418463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 949.447451] alg: No test for adler32 (adler32-zlib) [ 950.495131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.626058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 950.679925] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 951.949745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 952.024571] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 954.931066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 954.962586] LNet: Removed LNI 10.1.2.3@tcp [ 955.139403] LNet: 36205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.151956] LNetError: 36205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.173602] LNet: Removed LNI 192.168.204.42@tcp [ 956.134527] Key type .llcrypt unregistered [ 956.136195] Key type ._llcrypt unregistered [ 972.998886] Key type ._llcrypt registered [ 973.004633] Key type .llcrypt registered [ 975.002025] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:51:09 (1786035069) [ 976.049396] Key type .llcrypt unregistered [ 976.058631] Key type ._llcrypt unregistered [ 985.190120] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 985.494928] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 985.505147] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 985.513798] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 988.544985] Key type ._llcrypt registered [ 988.546211] Key type .llcrypt registered [ 988.983973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 989.021699] alg: No test for adler32 (adler32-zlib) [ 990.082337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.212657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 990.275430] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 991.997254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 992.035836] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 996.692095] LNet: 37706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.711841] LNetError: 37706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 996.720494] LNet: Removed LNI 10.1.2.3@tcp [ 996.742449] LNet: Removed LNI 192.168.204.42@tcp [ 997.869473] Key type .llcrypt unregistered [ 997.872356] Key type ._llcrypt unregistered [ 1015.340702] Key type ._llcrypt registered [ 1015.342286] Key type .llcrypt registered [ 1017.052938] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:51:52 (1786035112) [ 1018.409336] Key type .llcrypt unregistered [ 1018.416586] Key type ._llcrypt unregistered [ 1026.981252] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1027.303916] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1027.306087] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1027.310226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1030.133315] Key type ._llcrypt registered [ 1030.138836] Key type .llcrypt registered [ 1030.454858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1030.481633] alg: No test for adler32 (adler32-zlib) [ 1031.468506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1031.565382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 1031.616164] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1034.186956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1034.207861] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1037.366531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 1037.416343] LNet: Removed LNI 10.1.2.3@tcp [ 1037.431720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 1037.471969] LNet: Removed LNI 10.1.2.31@tcp [ 1037.491342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1037.533796] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1038.407735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 1038.439474] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1039.270976] LNet: 39638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1039.275873] LNetError: 39638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 1039.280648] LNet: Removed LNI 10.1.2.31@tcp [ 1039.311515] LNet: Removed LNI 10.1.2.3@tcp [ 1040.268193] Key type .llcrypt unregistered [ 1040.271211] Key type ._llcrypt unregistered [ 1058.584535] Key type ._llcrypt registered [ 1058.586167] Key type .llcrypt registered [ 1060.871709] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:52:35 (1786035155) [ 1061.360544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1061.377207] alg: No test for adler32 (adler32-zlib) [ 1062.522636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.653278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1062.690908] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1064.053075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1064.103710] LNet: 40595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.112908] LNetError: 40595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.148682] LNet: Removed LNI 192.168.204.42@tcp [ 1064.176996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1064.274882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.42@tcp [ 1064.308297] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1065.695680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.42@tcp [ 1065.746635] LNet: Removed LNI 192.168.204.42@tcp [ 1065.964894] LNet: 40731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.523785] Key type .llcrypt unregistered [ 1067.525530] Key type ._llcrypt unregistered [ 1092.005685] Key type ._llcrypt registered [ 1092.023944] Key type .llcrypt registered [ 1093.803888] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:53:08 (1786035188) [ 1096.933581] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg442-server [ 1099.742224] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:53:14 (1786035194) [ 1100.573295] Key type .llcrypt unregistered [ 1100.578215] Key type ._llcrypt unregistered [ 1113.488271] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1113.705222] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1113.709307] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1113.713507] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1117.060895] Key type ._llcrypt registered [ 1117.063337] Key type .llcrypt registered [ 1117.653819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1117.715058] alg: No test for adler32 (adler32-zlib) [ 1118.854967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1119.067332] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1120.438648] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 1127.248278] LNet: 42579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.260245] LNetError: 42579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.283694] LNet: Removed LNI 192.168.204.42@tcp1 [ 1127.312745] LNet: Removed LNI 10.1.2.3@tcp2 [ 1128.889596] Key type .llcrypt unregistered [ 1128.905022] Key type ._llcrypt unregistered [ 1149.156614] Key type ._llcrypt registered [ 1149.170783] Key type .llcrypt registered [ 1151.189095] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:54:06 (1786035246) [ 1151.762751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1151.792701] alg: No test for adler32 (adler32-zlib) [ 1152.887359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1152.984271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1153.141865] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1154.620284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1154.656808] LNet: 43571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.663578] LNetError: 43571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1154.699243] LNet: Removed LNI 192.168.204.42@tcp [ 1154.759674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1154.871433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 1154.916618] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1164.347354] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:54:19 (1786035259) [ 1164.912862] LNet: 44167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1164.936683] LNetError: 44167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1164.966057] LNet: Removed LNI 192.168.204.42@tcp [ 1166.046532] Key type .llcrypt unregistered [ 1166.048274] Key type ._llcrypt unregistered [ 1175.370671] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1175.510457] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1175.525914] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1175.544547] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1178.603682] Key type ._llcrypt registered [ 1178.607483] Key type .llcrypt registered [ 1178.940483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1178.962347] alg: No test for adler32 (adler32-zlib) [ 1180.056897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1180.178760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1180.227760] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1181.664411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1181.703144] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1185.117404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1185.159461] LNet: 44914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1185.169043] LNetError: 44914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1185.173943] LNet: Removed LNI 10.1.2.3@tcp [ 1185.186710] LNet: Removed LNI 192.168.204.42@tcp [ 1185.208632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1185.297777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1185.344708] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1185.354154] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1186.140918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1186.161731] LNet: 44941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.168866] LNetError: 44941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1186.174745] LNetError: 44941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1186.180250] LNet: Removed LNI 10.1.2.3@tcp [ 1186.194911] LNet: Removed LNI 192.168.204.42@tcp [ 1186.219284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1186.299957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1186.334032] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1186.338780] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1187.051611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1187.072598] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 1188.735583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1188.780810] LNet: 45104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1188.795549] LNetError: 45104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1188.810394] LNetError: 45104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1188.816152] LNet: Removed LNI 10.1.2.3@tcp [ 1188.818989] LNet: Removed LNI 192.168.204.42@tcp [ 1188.838099] LNet: Removed LNI 192.168.204.42@tcp2 [ 1188.855047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1188.944720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1188.974496] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1188.978054] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1188.982094] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 1199.161747] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:54:54 (1786035294) [ 1199.451305] LNet: 45701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1199.463928] LNetError: 45701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1199.478345] LNetError: 45701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1199.497050] LNet: Removed LNI 10.1.2.3@tcp [ 1199.498785] LNet: Removed LNI 192.168.204.42@tcp [ 1199.529425] LNet: Removed LNI 192.168.204.42@tcp2 [ 1200.189545] Key type .llcrypt unregistered [ 1200.192190] Key type ._llcrypt unregistered [ 1209.301962] Key type ._llcrypt registered [ 1209.308084] Key type .llcrypt registered [ 1209.718336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1209.751213] alg: No test for adler32 (adler32-zlib) [ 1210.781508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1210.856066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1210.905929] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1211.880723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1211.896977] LNet: 46100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1211.906816] LNetError: 46100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1211.928058] LNet: Removed LNI 192.168.204.42@tcp [ 1211.956154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1212.031307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1212.067127] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1212.650976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1212.697247] LNet: 46130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.713339] LNetError: 46130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1212.730629] LNet: Removed LNI 192.168.204.42@tcp [ 1212.769196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1212.864718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1212.896730] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1213.415583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1213.482591] LNet: Removed LNI 192.168.204.42@tcp [ 1213.566830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1213.664916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1213.700350] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1214.197794] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1214.409368] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1214.412198] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1214.415828] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1216.947741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1216.991509] LNet: 46238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1217.005762] LNet: 46238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1217.021301] LNetError: 46238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1217.028220] LNetError: 46238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1217.054075] LNet: Removed LNI 192.168.204.42@tcp [ 1217.098966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1217.212099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1217.270467] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1226.531775] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:55:21 (1786035321) [ 1226.934547] LNet: 46837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.942815] LNetError: 46837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.983812] LNet: Removed LNI 192.168.204.42@tcp [ 1227.974905] Key type .llcrypt unregistered [ 1227.984424] Key type ._llcrypt unregistered [ 1238.341668] Key type ._llcrypt registered [ 1238.364839] Key type .llcrypt registered [ 1238.725809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1238.762414] alg: No test for adler32 (adler32-zlib) [ 1240.032406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1240.141666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1240.173481] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1241.604482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1241.661158] LNet: 47240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.670867] LNetError: 47240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.699729] LNet: Removed LNI 192.168.204.42@tcp [ 1241.755123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1241.839765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1241.884768] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1254.663359] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:55:49 (1786035349) [ 1255.636799] LNet: 47833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.664305] LNetError: 47833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1255.693371] LNet: Removed LNI 192.168.204.42@tcp [ 1257.198946] Key type .llcrypt unregistered [ 1257.201727] Key type ._llcrypt unregistered [ 1266.457442] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1266.752473] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1266.763574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1266.779421] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1269.512103] Key type ._llcrypt registered [ 1269.514054] Key type .llcrypt registered [ 1269.733581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1269.762972] alg: No test for adler32 (adler32-zlib) [ 1270.724409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1270.840484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1270.884229] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1272.105603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1272.161816] LNet: 48287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.174901] LNetError: 48287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.204992] LNet: Removed LNI 192.168.204.42@tcp [ 1272.246284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1272.400399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1272.467747] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1283.427874] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:56:17 (1786035377) [ 1283.593457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1283.644629] LNet: 48880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1283.650053] LNetError: 48880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1283.677318] LNet: Removed LNI 192.168.204.42@tcp [ 1283.699941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1283.835374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1297.978457] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:56:31 (1786035391) [ 1298.653104] LNet: 49467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1300.695068] Key type .llcrypt unregistered [ 1300.696451] Key type ._llcrypt unregistered [ 1318.079853] Key type ._llcrypt registered [ 1318.082765] Key type .llcrypt registered [ 1318.310928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1318.347504] alg: No test for adler32 (adler32-zlib) [ 1319.373979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1319.455602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1319.489739] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1320.265269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1320.286575] LNet: 49873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1320.291577] LNetError: 49873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1320.299670] LNet: Removed LNI 192.168.204.42@tcp [ 1320.325641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1320.399121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1320.429956] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1330.230392] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:57:05 (1786035425) [ 1330.426880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1330.482675] LNet: 50461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1330.505375] LNetError: 50461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.543099] LNet: Removed LNI 192.168.204.42@tcp [ 1330.624097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1330.830428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1338.140837] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:57:14 (1786035434) [ 1340.570791] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1343.092694] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:57:18 (1786035438) [ 1345.334908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1347.833204] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:57:23 (1786035443) [ 1349.680542] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1351.979299] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:57:27 (1786035447) [ 1354.520718] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1357.683138] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:57:32 (1786035452) [ 1359.921460] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1362.210926] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:57:37 (1786035457) [ 1362.362274] LNet: 52513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.558580] Key type .llcrypt unregistered [ 1363.561930] Key type ._llcrypt unregistered [ 1373.560625] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1373.752581] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1373.761928] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1373.774124] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1377.215873] Key type ._llcrypt registered [ 1377.221202] Key type .llcrypt registered [ 1377.689719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1377.734285] alg: No test for adler32 (adler32-zlib) [ 1378.868594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.036929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1379.068483] LNet: Added LNI 192.168.204.42@tcp [8/512/0/180] [ 1379.076166] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1379.082676] LNet: 52846:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1379.297432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.380701] LNet: 52865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1379.402647] LNetError: 52865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1379.416093] LNet: Removed LNI 10.1.2.3@tcp [ 1379.417784] LNetError: 52865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1379.447090] LNet: Removed LNI 192.168.204.42@tcp [ 1379.525501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.730533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1379.811976] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1379.979863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1380.043487] LNet: 52889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1380.077741] LNetError: 52889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1380.137400] LNet: Removed LNI 192.168.204.42@tcp [ 1380.222616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1380.372064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1380.426731] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1380.429882] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1380.437821] LNet: 52897:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1380.686077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1380.771684] LNet: 52915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1380.792672] LNetError: 52915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1380.804523] LNet: Removed LNI 10.1.2.3@tcp [ 1380.812653] LNetError: 52915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1380.863367] LNet: Removed LNI 192.168.204.42@tcp [ 1380.953396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.120048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1381.191429] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1381.202507] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1381.210790] LNet: 52923:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1381.554810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1381.606146] LNet: 52950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1381.621812] LNetError: 52950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1381.636462] LNet: Removed LNI 10.1.2.3@tcp [ 1381.643503] LNetError: 52950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1381.691275] LNet: Removed LNI 192.168.204.42@tcp [ 1381.740898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.887683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1381.944900] LNet: Added LNI 192.168.204.42@tcp [8/256/0/182] [ 1381.959291] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1382.183966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1382.240082] LNet: 52978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.256455] LNetError: 52978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1382.279871] LNet: Removed LNI 10.1.2.3@tcp [ 1382.283361] LNetError: 52978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1382.306447] LNet: Removed LNI 192.168.204.42@tcp [ 1382.351296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.565174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1382.623859] LNet: Added LNI 192.168.204.42@tcp [8/256/0/182] [ 1382.645513] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1382.647603] LNet: 52986:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1382.783966] LNet: Removed LNI 10.1.2.3@tcp [ 1382.804609] LNet: Removed LNI 192.168.204.42@tcp [ 1383.987842] Key type .llcrypt unregistered [ 1383.991988] Key type ._llcrypt unregistered [ 1400.463567] Key type ._llcrypt registered [ 1400.465386] Key type .llcrypt registered [ 1402.658170] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 12:58:17 (1786035497) [ 1404.658295] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1406.543132] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:58:22 (1786035502) [ 1407.214209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1407.283021] alg: No test for adler32 (adler32-zlib) [ 1408.293834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1408.411942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1408.443259] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1409.693139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.42 [ 1409.722040] LNet: 54255:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1409.741456] LNet: Removed LNI 192.168.204.42@tcp [ 1421.344594] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:58:36 (1786035516) [ 1421.518165] LNet: 54817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1421.536620] LNetError: 54817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1421.568473] LNet: Removed LNI 192.168.204.42@tcp [ 1422.907365] Key type .llcrypt unregistered [ 1422.914081] Key type ._llcrypt unregistered [ 1436.595111] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1437.292981] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1437.318989] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1437.343782] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1439.642679] Key type ._llcrypt registered [ 1439.644913] Key type .llcrypt registered [ 1439.800578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1439.814294] alg: No test for adler32 (adler32-zlib) [ 1441.404542] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1451.007640] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:59:06 (1786035546) [ 1451.675312] LNet: 55737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.694845] LNetError: 55737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1451.713169] LNet: Removed LNI 10.1.2.3@tcp [ 1452.885581] Key type .llcrypt unregistered [ 1452.890476] Key type ._llcrypt unregistered [ 1462.739181] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1463.170229] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1463.179197] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1463.190737] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1466.405274] Key type ._llcrypt registered [ 1466.408467] Key type .llcrypt registered [ 1466.789882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1466.843988] alg: No test for adler32 (adler32-zlib) [ 1468.007970] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1476.899848] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:59:32 (1786035572) [ 1479.760891] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1481.656351] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:59:37 (1786035577) [ 1481.983309] LNet: 56927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.997670] LNetError: 56927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1482.018535] LNet: Removed LNI 10.1.2.3@tcp [ 1483.086628] Key type .llcrypt unregistered [ 1483.090449] Key type ._llcrypt unregistered [ 1491.168047] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1491.483718] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1491.489946] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1491.492155] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1493.386361] Key type ._llcrypt registered [ 1493.399481] Key type .llcrypt registered [ 1493.844251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1493.878980] alg: No test for adler32 (adler32-zlib) [ 1494.909848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1495.009328] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1505.614528] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 13:00:01 (1786035601) [ 1506.512553] LNet: 58169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.521654] LNetError: 58169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1506.539864] LNet: Removed LNI 10.1.2.3@tcp [ 1507.308159] Key type .llcrypt unregistered [ 1507.314475] Key type ._llcrypt unregistered [ 1515.352624] Key type ._llcrypt registered [ 1515.358927] Key type .llcrypt registered [ 1515.658139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1515.680780] alg: No test for adler32 (adler32-zlib) [ 1516.611859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1516.666255] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1528.816601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1530.715187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_interrupt -r 1 [ 1530.807554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_interrupt -r 1 [ 1530.891824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1530.952895] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1531.040622] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1531.425902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1531.474291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1531.529424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1531.583539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1531.627843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1531.665663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1531.709101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1531.748667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1531.776496] LNet: 58612:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1531.807043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1531.874687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1531.919967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1531.991559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1532.026763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1532.073313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1532.140628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1532.859562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_dropped -r 1 [ 1532.938723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_dropped -r 1 [ 1533.030503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1533.083494] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1533.101725] LNet: Skipped 1 previous similar message [ 1533.347916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1533.393518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1533.429418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1533.479501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1533.527437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1533.555957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1533.596743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1533.646392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1533.684886] LNet: 58721:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1533.718053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1533.751442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1533.775749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1533.802676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1533.826265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1533.852877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1533.889334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1534.413739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_aborted -r 1 [ 1534.444288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_aborted -r 1 [ 1534.474484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1534.490666] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1534.671868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1534.696916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1534.733305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1534.783467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1534.852400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1534.901953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1534.953428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1534.996782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1535.016633] LNet: 58830:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1535.043897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1535.076750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1535.109480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1535.152424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1535.212787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1535.279331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1535.334592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1536.159992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_no_route -r 1 [ 1536.225067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_no_route -r 1 [ 1536.301341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1536.741566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1536.844774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1536.938118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1537.047563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1537.139537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1537.219602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1537.303336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1537.393856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1537.459100] LNet: 58940:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1537.538024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1537.632024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1537.791551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1537.875547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1537.996468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1538.107669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1538.172065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1539.272844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_timeout -r 1 [ 1539.348523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_timeout -r 1 [ 1539.408597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1539.432718] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1539.443451] LNet: Skipped 1 previous similar message [ 1539.814525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1539.843573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1539.897724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1539.946596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1540.003907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1540.054457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1540.105528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1540.150881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1540.189139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1540.253415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1540.293523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1540.333853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1540.394435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1540.454756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1540.503769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1541.117407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_error -r 1 [ 1541.155815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_error -r 1 [ 1541.192126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1541.627270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1541.676473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1541.745329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1541.845884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1541.923458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1541.999849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1542.065745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1542.129786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1542.164674] LNet: 59158:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1542.174795] LNet: 59158:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1542.212686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1542.297910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1542.411911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1542.519255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1542.618224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1542.714424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1542.783959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1543.555900] LNet: 59203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.560951] LNetError: 59203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.606647] LNet: Removed LNI 192.168.204.42@tcp [ 1544.246594] Key type .llcrypt unregistered [ 1544.248617] Key type ._llcrypt unregistered [ 1572.578051] Key type ._llcrypt registered [ 1572.581831] Key type .llcrypt registered [ 1574.509194] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 13:01:09 (1786035669) [ 1575.936056] Key type .llcrypt unregistered [ 1575.946367] Key type ._llcrypt unregistered [ 1585.345873] Key type ._llcrypt registered [ 1585.347887] Key type .llcrypt registered [ 1585.569981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1585.605593] alg: No test for adler32 (adler32-zlib) [ 1586.609934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1586.667560] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1595.981714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1611.359076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.424979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1611.484548] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1613.717279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_interrupt -r 1 [ 1613.763103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 1 [ 1613.825775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_interrupt -r 1 [ 1613.868737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 1 [ 1613.942027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_interrupt -r 1 [ 1614.022306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 1 [ 1614.078808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_interrupt -r 1 [ 1614.136776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 1 [ 1614.211386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1614.241591] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1614.248650] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1614.563723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1614.622735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1614.668767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1614.724991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1614.780662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1614.841065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1614.911175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1614.976766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1615.016348] LNet: 60691:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1615.047488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1615.100989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1615.156715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1615.210243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1615.267562] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1615.277225] LNet: Skipped 3 previous similar messages [ 1615.283503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1615.324060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1615.371835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1616.185534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_dropped -r 1 [ 1616.239847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 1 [ 1616.297979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_dropped -r 1 [ 1616.367963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 1 [ 1616.431674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_dropped -r 1 [ 1616.509949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 1 [ 1616.591198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_dropped -r 1 [ 1616.685596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 1 [ 1616.767211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1616.821655] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1617.239707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1617.323362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1617.392929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1617.476854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1617.549398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1617.590712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1617.673795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1617.732297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1617.759775] LNet: 60812:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1617.788888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1617.859966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1617.908923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1617.982418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1618.035694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1618.091766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1618.154701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1618.901733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_aborted -r 1 [ 1618.973807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 1 [ 1619.035985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_aborted -r 1 [ 1619.090008] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 1 [ 1619.154534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_aborted -r 1 [ 1619.210668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 1 [ 1619.255514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_aborted -r 1 [ 1619.310204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 1 [ 1619.347912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1619.377542] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1619.382179] LNet: Skipped 2 previous similar messages [ 1619.583600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1619.610452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1619.631803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1619.662718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1619.685333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1619.712389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1619.756524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1619.805879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1619.829279] LNet: 60934:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1619.857713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1619.900872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1619.925899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1619.945413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1619.970485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1619.990848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1620.015448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1620.493772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_no_route -r 1 [ 1620.527021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 1 [ 1620.551711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_no_route -r 1 [ 1620.579693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 1 [ 1620.604784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_no_route -r 1 [ 1620.632573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 1 [ 1620.661634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_no_route -r 1 [ 1620.690593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 1 [ 1620.715571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1621.060984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1621.104181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1621.169253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1621.235297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1621.327614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1621.412847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1621.499759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1621.621062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1621.737761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1621.838800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1621.921924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1621.979846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1622.051426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1622.135906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1622.227227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1623.157926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_timeout -r 1 [ 1623.249637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 1 [ 1623.327864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_timeout -r 1 [ 1623.391687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 1 [ 1623.464685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_timeout -r 1 [ 1623.522328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 1 [ 1623.599179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_timeout -r 1 [ 1623.661653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 1 [ 1623.732076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1623.781812] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1623.791926] LNet: Skipped 5 previous similar messages [ 1624.090797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1624.146253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1624.198906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1624.254425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1624.310396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1624.370140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1624.434259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1624.492571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1624.513971] LNet: 61176:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1624.524103] LNet: 61176:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1624.561651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1624.609916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1624.663158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1624.713773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1624.782424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1624.836916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1624.883682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1625.660682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_error -r 1 [ 1625.735558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_error -r 1 [ 1625.791657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_error -r 1 [ 1625.844680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_error -r 1 [ 1625.889462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_error -r 1 [ 1625.938846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_error -r 1 [ 1625.987208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_error -r 1 [ 1626.038625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_error -r 1 [ 1626.093668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1626.323474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1626.354350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1626.384630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1626.413160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1626.464779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1626.515561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1626.554034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1626.613665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1626.655934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1626.718809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1626.780669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1626.842956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1626.899754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1626.963938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1627.031785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1633.776193] LNet: 61378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.788075] LNetError: 61378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.798144] LNet: Removed LNI 192.168.204.42@tcp [ 1634.855233] LNet: Removed LNI 192.168.204.42@tcp1 [ 1635.383685] Key type .llcrypt unregistered [ 1635.387904] Key type ._llcrypt unregistered [ 1661.587818] Key type ._llcrypt registered [ 1661.589882] Key type .llcrypt registered [ 1663.330859] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 13:02:38 (1786035758) [ 1665.117465] Key type .llcrypt unregistered [ 1665.122375] Key type ._llcrypt unregistered [ 1674.410486] Key type ._llcrypt registered [ 1674.419565] Key type .llcrypt registered [ 1674.662571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1674.676833] alg: No test for adler32 (adler32-zlib) [ 1675.818126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1675.849620] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1687.239317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1687.918542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_dropped -r 1 [ 1687.950340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_dropped -r 1 [ 1687.992882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1688.009964] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1688.208743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1688.240348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1688.266933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1688.295767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1688.324871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1688.351618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1688.382591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1688.408395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1688.431900] LNet: 62632:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1688.450471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1688.486514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1688.510704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1688.534720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1688.559766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1688.588889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1688.618715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1689.175299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_error -r 1 [ 1689.219346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_error -r 1 [ 1689.291653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1689.315607] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1689.522392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1689.575810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1689.620295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1689.647191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1689.680842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1689.702971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1689.736724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1689.782148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1689.813816] LNet: 62742:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1689.841626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1689.875622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1689.901807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1689.925636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1689.950516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1689.980886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1690.013942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1690.513645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_timeout -r 1 [ 1690.545190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_timeout -r 1 [ 1690.569401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1690.587700] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1690.788889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1690.820596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1690.848519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1690.872138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1690.902426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1690.930231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1690.960870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1690.988770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1691.008139] LNet: 62851:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1691.035845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1691.068588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1691.098219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1691.127213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1691.150273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1691.180928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1691.213838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1691.769842] LNet: 62896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.782118] LNetError: 62896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.801524] LNet: Removed LNI 192.168.204.42@tcp [ 1692.554625] Key type .llcrypt unregistered [ 1692.558142] Key type ._llcrypt unregistered [ 1720.498913] Key type ._llcrypt registered [ 1720.501157] Key type .llcrypt registered [ 1722.477876] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 13:03:37 (1786035817) [ 1723.777374] Key type .llcrypt unregistered [ 1723.783044] Key type ._llcrypt unregistered [ 1734.317735] Key type ._llcrypt registered [ 1734.328767] Key type .llcrypt registered [ 1734.776666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1734.849706] alg: No test for adler32 (adler32-zlib) [ 1735.927264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1735.947142] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1745.645200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1759.180271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.300388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1759.401054] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1762.190777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_dropped -r 1 [ 1762.236181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 1 [ 1762.275296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_dropped -r 1 [ 1762.329961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 1 [ 1762.388735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_dropped -r 1 [ 1762.441374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 1 [ 1762.482251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_dropped -r 1 [ 1762.547956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 1 [ 1762.643842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1762.680657] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1762.688661] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 1763.077240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1763.126848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1763.166847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1763.213553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1763.237901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1763.264258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1763.302867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1763.335199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1763.360441] LNet: 64372:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1763.380316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1763.414578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1763.450866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1763.487971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1763.522402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1763.572833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1763.607452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1763.643149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1764.582859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_error -r 1 [ 1764.690623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_error -r 1 [ 1764.833570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_error -r 1 [ 1764.953905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_error -r 1 [ 1765.006890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_error -r 1 [ 1765.035129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_error -r 1 [ 1765.059976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_error -r 1 [ 1765.082669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_error -r 1 [ 1765.108872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1765.121135] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1765.124217] LNet: Skipped 2 previous similar messages [ 1765.588201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1765.658193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1765.711296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1765.773628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1765.840913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1765.925428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1766.056944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1766.157463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1766.218744] LNet: 64495:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1766.250910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1766.311164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1766.342358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1766.378644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1766.406885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1766.432948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1766.468462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1766.514742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1767.559408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_timeout -r 1 [ 1767.637299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_timeout -r 1 [ 1767.705713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_timeout -r 1 [ 1767.784065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_timeout -r 1 [ 1767.869425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_timeout -r 1 [ 1767.961514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_timeout -r 1 [ 1768.052316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_timeout -r 1 [ 1768.116086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_timeout -r 1 [ 1768.178704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1768.204570] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1768.755772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1768.823420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1768.900840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1768.983649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1769.075744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1769.158486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1769.230906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1769.343280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1769.489219] LNet: 64619:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1769.573104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1769.690668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1769.863965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1769.958891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1770.090465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1770.216608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1770.290219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1770.366570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1777.799763] LNet: 64701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.816777] LNetError: 64701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.830614] LNet: Removed LNI 192.168.204.42@tcp [ 1777.840958] LNet: Removed LNI 192.168.204.42@tcp1 [ 1778.679624] Key type .llcrypt unregistered [ 1778.685122] Key type ._llcrypt unregistered [ 1807.026798] Key type ._llcrypt registered [ 1807.029068] Key type .llcrypt registered [ 1809.175534] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 13:05:03 (1786035903) [ 1811.289717] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1813.892446] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 13:05:09 (1786035909) [ 1815.422546] Key type .llcrypt unregistered [ 1815.425829] Key type ._llcrypt unregistered [ 1825.420361] Key type ._llcrypt registered [ 1825.433408] Key type .llcrypt registered [ 1825.832893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1825.875771] alg: No test for adler32 (adler32-zlib) [ 1826.862447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1826.938293] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1838.402149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1839.612267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e network_timeout -r 1 [ 1839.647496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e network_timeout -r 1 [ 1839.723607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1839.767354] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1840.137125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1840.177695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1840.222776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1840.281819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1840.293446] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1840.308669] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1840.340654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1840.392364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1840.453169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1840.512148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1840.541197] LNet: 66256:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1840.562417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1840.608795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1840.655579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1840.711642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1840.785197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1840.843718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1841.429157] LNet: 66299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.438044] LNetError: 66299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.460822] LNet: Removed LNI 192.168.204.42@tcp [ 1842.172539] Key type .llcrypt unregistered [ 1842.178957] Key type ._llcrypt unregistered [ 1873.355286] Key type ._llcrypt registered [ 1873.356800] Key type .llcrypt registered [ 1873.833759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1873.880840] alg: No test for adler32 (adler32-zlib) [ 1875.032577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1875.087756] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1886.277477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1898.826662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.884601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1898.926366] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1901.398360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e network_timeout -r 1 [ 1901.450145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e network_timeout -r 1 [ 1901.507214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e network_timeout -r 1 [ 1901.560873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e network_timeout -r 1 [ 1901.631394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e network_timeout -r 1 [ 1901.720198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e network_timeout -r 1 [ 1901.776665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e network_timeout -r 1 [ 1901.860348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e network_timeout -r 1 [ 1901.927633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1901.957917] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1902.275937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1902.322130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1902.377605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1902.435524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1902.490146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1902.540756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1902.577768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1902.656335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1902.693238] LNet: 67199:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1902.725336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1902.751397] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1902.753565] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 1902.763899] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1902.774107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1902.778410] LNet: Skipped 1 previous similar message [ 1902.863948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1902.932966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1903.008895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1903.100632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1910.235537] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1910.988032] LNet: 67277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.997606] LNetError: 67277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1912.031889] LNet: Removed LNI 192.168.204.42@tcp [ 1912.033677] LNetError: 67277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1912.073145] LNet: Removed LNI 192.168.204.42@tcp1 [ 1912.767652] Key type .llcrypt unregistered [ 1912.770670] Key type ._llcrypt unregistered [ 1942.122635] Key type ._llcrypt registered [ 1942.125475] Key type .llcrypt registered [ 1944.845237] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 13:07:19 (1786036039) [ 1945.319438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1945.354636] alg: No test for adler32 (adler32-zlib) [ 1946.520666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.709653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1946.807939] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1948.304725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1948.359069] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1950.474307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1950.638658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1950.769755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1950.846351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1950.916091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1950.976940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1951.044240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.42@tcp [ 1951.064794] LNetError: 68380:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 1951.775312] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1951.775792] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1953.826338] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1955.021685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1957.929061] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1959.468168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1959.604250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1959.874391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1959.996372] LNet: 68487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.019960] LNetError: 68487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.049339] LNet: Removed LNI 192.168.204.42@tcp [ 1960.081596] LNet: Removed LNI 192.168.204.42@tcp1 [ 1960.222818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1960.409282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1960.503050] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1962.754490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1962.828121] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1964.451815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1964.631515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1964.768552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1964.900521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1964.994882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1965.117191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1965.229974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1965.350038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.42@tcp [ 1965.435317] LNetError: 68760:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 1965.478288] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1968.222281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1971.649686] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1971.674725] LNet: Skipped 1 previous similar message [ 1977.172267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1977.319898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1977.437182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1992.769947] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 13:08:08 (1786036088) [ 1993.546703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1993.580924] LNet: 69458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.599327] LNetError: 69458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1993.610321] LNetError: 69458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1993.618974] LNet: Removed LNI 192.168.204.42@tcp [ 1993.645838] LNet: Removed LNI 192.168.204.42@tcp1 [ 1993.676665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.818545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1993.851449] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1994.894531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1994.928552] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1996.434676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1996.536670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1996.588533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1996.641364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1996.694315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1996.724580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1996.757365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.42@tcp --health 0 [ 1996.831172] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.42@tcp [ 1997.002692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1997.860573] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1997.877076] LNet: Skipped 1 previous similar message [ 2013.100492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2013.381815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2013.434948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2013.500177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.42@tcp --health 500 [ 2013.699446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2019.538757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2019.654500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2019.684901] LNet: 69942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.697401] LNetError: 69942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.708417] LNetError: 69942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2019.734613] LNet: Removed LNI 192.168.204.42@tcp [ 2019.770849] LNet: Removed LNI 192.168.204.42@tcp1 [ 2019.832567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2019.898844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2019.921608] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2020.611946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2020.628434] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 2021.626278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2021.670158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2021.724293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 2021.749536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 2021.783109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 2021.815277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2021.838833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 2021.861803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.42@tcp --health 0 [ 2022.943715] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 2022.965104] LNet: Skipped 3 previous similar messages [ 2023.261876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2032.976402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2033.011443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2033.077528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2044.767721] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 13:08:59 (1786036139) [ 2045.255373] LNet: 70979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.259993] LNetError: 70979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2045.264809] LNetError: 70979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2045.269613] LNet: Removed LNI 192.168.204.42@tcp [ 2045.286754] LNet: Removed LNI 192.168.204.42@tcp1 [ 2046.359455] Key type .llcrypt unregistered [ 2046.364564] Key type ._llcrypt unregistered [ 2054.856397] Key type ._llcrypt registered [ 2054.864100] Key type .llcrypt registered [ 2055.195762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2055.233164] alg: No test for adler32 (adler32-zlib) [ 2056.389686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2056.453797] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2070.669887] Lustre: DEBUG MARKER: Initial discovery [ 2070.746422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.142@tcp [ 2074.749703] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 2074.881835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 2074.999762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.142@tcp [ 2075.060285] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 2075.101815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2075.203272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.142@tcp -r 1 -m GET -l 3 [ 2075.312610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.142@tcp -r 1 -m GET -e local_error [ 2075.406047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.142@tcp -r 1 -m PUT -l 6 [ 2075.944221] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 2077.946560] Lustre: DEBUG MARKER: Force oleg442-client.virtnet to discover 192.168.204.142@tcp (in background) [ 2078.150678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.142@tcp [ 2080.184911] Lustre: DEBUG MARKER: Force oleg442-server to discover 192.168.204.42@tcp [ 2081.247491] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 2083.697825] Lustre: DEBUG MARKER: Wait for 71739 [ 2084.902902] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 2085.594332] Lustre: DEBUG MARKER: Finished wait on 71739 [ 2086.382720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 2086.453146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2086.984789] LNet: 72200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.993331] LNetError: 72200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2087.012726] LNet: Removed LNI 192.168.204.42@tcp [ 2087.696669] Key type .llcrypt unregistered [ 2087.702865] Key type ._llcrypt unregistered [ 2111.675854] Key type ._llcrypt registered [ 2111.679292] Key type .llcrypt registered [ 2113.333424] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 13:10:08 (1786036208) [ 2114.222210] Key type .llcrypt unregistered [ 2114.227800] Key type ._llcrypt unregistered [ 2124.052753] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2124.207213] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2124.220312] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2124.225435] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2127.165538] Key type ._llcrypt registered [ 2127.170314] Key type .llcrypt registered [ 2127.365242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2127.387930] alg: No test for adler32 (adler32-zlib) [ 2128.367635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2128.430255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2128.489339] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2129.265288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2129.284351] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2140.457885] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 13:10:35 (1786036235) [ 2140.976230] LNet: 74278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2140.988119] LNetError: 74278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2141.000417] LNet: Removed LNI 10.1.2.3@tcp [ 2141.029967] LNet: Removed LNI 192.168.204.42@tcp [ 2142.009405] Key type .llcrypt unregistered [ 2142.019882] Key type ._llcrypt unregistered [ 2153.401140] Key type ._llcrypt registered [ 2153.405514] Key type .llcrypt registered [ 2153.614550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2153.692641] alg: No test for adler32 (adler32-zlib) [ 2154.690323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.816534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2154.874251] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2156.388832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2156.462359] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2158.374863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.42@tcp [ 2158.464452] LNetError: Unexpected error -22 connecting to 192.168.204.42@tcp at host 192.168.204.42:988 [ 2168.298099] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 13:11:03 (1786036263) [ 2168.836773] LNet: 75415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.858495] LNetError: 75415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2169.887181] LNet: Removed LNI 10.1.2.3@tcp [ 2169.889894] LNetError: 75415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2169.920403] LNet: Removed LNI 192.168.204.42@tcp [ 2171.022502] Key type .llcrypt unregistered [ 2171.034302] Key type ._llcrypt unregistered [ 2179.513160] Key type ._llcrypt registered [ 2179.519421] Key type .llcrypt registered [ 2179.934052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2179.960113] alg: No test for adler32 (adler32-zlib) [ 2180.934756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.008214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2181.036265] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 2182.446276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2182.518594] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2184.816061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.42@tcp1 --nid 192.168.204.42@tcp2 [ 2192.211775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp1 192.168.204.42@tcp2 [ 2192.256651] LNetError: 76255:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.42@tcp1 to 192.168.204.42@tcp2 - Route table may be misconfigured [ 2192.303496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp2 192.168.204.42@tcp1 [ 2200.675692] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 13:11:36 (1786036296) [ 2200.885950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2200.956756] LNet: 76828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.971759] LNetError: 76828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2200.986354] LNet: Removed LNI 192.168.204.42@tcp1 [ 2202.026617] LNet: Removed LNI 192.168.204.42@tcp2 [ 2202.136370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2202.296376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2202.353977] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2204.493868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2204.530499] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 2206.830803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2206.973227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.42@tcp [ 2207.096424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp1 192.168.204.42@tcp1 [ 2207.243113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.42@tcp -d 192.168.204.42@tcp -e network_timeout [ 2207.336645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -e network_timeout [ 2207.443968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -e network_timeout [ 2207.514539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -e network_timeout [ 2207.625992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2207.675786] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 2207.725055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2207.955348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2208.201140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2208.352150] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 2220.106195] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 13:11:55 (1786036315) [ 2220.192221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2220.221344] LNet: 77684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.235777] LNetError: 77684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.249099] LNetError: 77684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2220.271218] LNet: Removed LNI 192.168.204.42@tcp [ 2221.293150] LNet: Removed LNI 192.168.204.42@tcp1 [ 2221.325430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2221.415423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 2221.721877] LNet: 77734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.211236] Key type .llcrypt unregistered [ 2222.219063] Key type ._llcrypt unregistered [ 2238.573601] Key type ._llcrypt registered [ 2238.577336] Key type .llcrypt registered [ 2239.979202] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 13:12:15 (1786036335) [ 2240.806954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2240.824628] alg: No test for adler32 (adler32-zlib) [ 2241.783537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.559947] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 2245.325541] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 13:12:21 (1786036341) [ 2245.527933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2245.550978] LNet: 78834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2245.574606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2245.630407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2245.653129] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2246.473583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2246.513514] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 2247.450069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2247.481663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp1 [ 2247.516628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp1 [ 2255.500573] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 13:12:30 (1786036350) [ 2257.508837] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg442-server [ 2259.988181] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 13:12:35 (1786036355) [ 2262.528108] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg442-server [ 2264.731235] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 13:12:40 (1786036360) [ 2268.144404] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg442-server [ 2270.897742] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 13:12:45 (1786036365) [ 2273.674427] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg442-server [ 2276.370196] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 13:12:51 (1786036371) [ 2278.342862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg442-server [ 2280.240846] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 13:12:55 (1786036375) [ 2283.408439] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg442-server [ 2286.268186] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 13:13:01 (1786036381) [ 2288.847706] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg442-server [ 2291.822848] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 13:13:06 (1786036386) [ 2294.595388] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg442-server [ 2296.514712] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 13:13:12 (1786036392) [ 2298.528709] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg442-server [ 2301.583287] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 13:13:16 (1786036396) [ 2301.767144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2301.810455] LNet: 82707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.820862] LNetError: 82707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2301.835142] LNet: Removed LNI 192.168.204.42@tcp [ 2302.886688] LNet: Removed LNI 192.168.204.42@tcp1 [ 2302.967938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2303.058132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2303.080852] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2303.984527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 2304.127980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2304.374914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2304.413954] LNet: 82861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.433531] LNetError: 82861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2304.444345] LNetError: 82861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2304.474104] LNet: Removed LNI 192.168.204.42@tcp [ 2304.513097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.576649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 2304.609899] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2305.683328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2307.217119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2307.255876] LNet: 83016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2307.265235] LNetError: 83016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2307.279750] LNet: Removed LNI 192.168.204.42@tcp [ 2307.332692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2307.431723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2307.499499] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2308.483496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 2308.623737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2308.913108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2310.062977] LNet: Removed LNI 192.168.204.42@tcp [ 2310.137748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.260873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 2310.318848] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2311.352812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2311.577676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.617126] LNet: 83308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.625363] LNet: 83308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2311.654145] LNetError: 83308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2311.663272] LNetError: 83308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2312.677823] LNet: Removed LNI 192.168.204.42@tcp [ 2312.698350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.819493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2312.860180] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2314.086770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 2314.196977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2314.512581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2315.557228] LNet: Removed LNI 192.168.204.42@tcp [ 2315.603072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2315.708294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 2315.745847] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2316.832576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2317.195405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2317.266469] LNetError: 83602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2317.274062] LNetError: 83602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2317.299850] LNet: Removed LNI 192.168.204.42@tcp [ 2317.352396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2317.469099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2317.509174] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2318.539385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 2318.637749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2318.901769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2319.982849] LNet: Removed LNI 192.168.204.42@tcp [ 2320.040872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2320.181988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 2320.222404] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2321.321887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2321.675890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2321.712157] LNet: 83894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2321.726887] LNet: 83894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2322.790921] LNet: Removed LNI 192.168.204.42@tcp [ 2322.880497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.034307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2323.110817] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2324.613905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 2324.718780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2325.245874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2325.301594] LNetError: 84042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.310297] LNetError: 84042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2325.357202] LNet: Removed LNI 192.168.204.42@tcp [ 2325.451796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2325.619817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 2325.726622] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2327.205867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2327.621812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2328.743787] LNet: Removed LNI 192.168.204.42@tcp [ 2328.818251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.929267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2329.013723] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2330.299931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 2330.411554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2331.051903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2332.136681] LNet: Removed LNI 192.168.204.42@tcp [ 2332.174818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2332.276318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 2332.332227] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2333.538294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2333.916367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.013278] LNet: Removed LNI 192.168.204.42@tcp [ 2335.068074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.181182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2335.254646] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2336.357787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 2336.498917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2336.593405] LNet: Removed LNI 192.168.204.42@tcp [ 2336.668907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.809652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2336.847843] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2337.947168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 2338.018454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2338.036961] LNet: 84751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2338.041735] LNet: 84751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2338.060088] LNet: Removed LNI 192.168.204.42@tcp [ 2338.108427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2338.211582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 2338.343294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2338.420295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2338.519920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 2338.636986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2338.691729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2338.787573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 2338.934822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.005425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2339.080686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 2339.133946] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2340.184373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.227812] LNet: Removed LNI 192.168.204.42@tcp [ 2340.258908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.314926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2348.677040] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 13:14:04 (1786036444) [ 2349.128056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2349.210421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2349.332818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 2349.402315] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 2351.730834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2351.782591] LNetError: 85641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2351.790407] LNetError: 85641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2351.812995] LNet: Removed LNI 192.168.204.42@tcp231 [ 2351.869079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2352.048654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2352.101324] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 2352.116759] LNet: 85649:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2352.896327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2352.940527] LNet: Removed LNI 192.168.204.42@tcp231 [ 2352.985869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 2353.011228] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 2354.189327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.228589] LNet: Removed LNI 192.168.204.42@tcp231 [ 2354.252661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2354.319312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2354.347257] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/0] [ 2354.355587] LNet: 85802:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2354.848497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2354.885707] LNet: Removed LNI 192.168.204.42@tcp231 [ 2354.918351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 2354.955980] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/0] [ 2356.862509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2356.915437] LNet: Removed LNI 192.168.204.42@tcp231 [ 2356.950679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2357.079723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2357.126646] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/60] [ 2357.143931] LNet: 85953:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2357.931315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2357.997300] LNet: Removed LNI 192.168.204.42@tcp231 [ 2358.033414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 2358.088601] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/60] [ 2359.931212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2359.981391] LNet: Removed LNI 192.168.204.42@tcp231 [ 2360.023672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.101887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2360.150624] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 2360.163288] LNet: 86105:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2360.698478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2360.732933] LNet: Removed LNI 192.168.204.42@tcp231 [ 2360.778398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 2360.873380] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 2370.996925] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 13:14:26 (1786036466) [ 2371.318326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2371.348041] LNet: 86816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2371.358051] LNet: 86816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 2371.391125] LNet: Removed LNI 192.168.204.42@tcp231 [ 2371.432246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.543941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 2371.585803] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 2372.694067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2372.783471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2372.823783] LNet: Removed LNI 192.168.204.42@tcp232 [ 2372.843872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 2372.887437] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 2373.967511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2374.030322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2374.110083] LNet: Removed LNI 192.168.204.42@tcp232 [ 2374.127571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 2374.169282] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 2375.546811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2375.611981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2375.664818] LNet: Removed LNI 192.168.204.42@tcp232 [ 2375.700666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 2375.739173] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 2376.760321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2376.829260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2376.883263] LNet: Removed LNI 192.168.204.42@tcp232 [ 2376.912019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2385.817908] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 13:14:41 (1786036481) [ 2388.356632] Key type .llcrypt unregistered [ 2388.361882] Key type ._llcrypt unregistered [ 2399.090727] Key type ._llcrypt registered [ 2399.095436] Key type .llcrypt registered [ 2399.579485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2399.607458] alg: No test for adler32 (adler32-zlib) [ 2400.697095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2400.751973] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2413.300359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 2427.329844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2427.417838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2427.467519] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 2429.566821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_interrupt -r 2 [ 2429.624543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 2 [ 2429.689732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_interrupt -r 2 [ 2429.783350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 2 [ 2429.853249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_interrupt -r 2 [ 2429.941242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 2 [ 2430.023671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_interrupt -r 2 [ 2430.093844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 2 [ 2430.171254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 2430.218260] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 2430.234601] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 2430.563349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2430.610587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2430.672810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2430.754406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2430.841968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2430.923920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2431.001396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2431.066480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2431.105964] LNet: 88615:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2431.147316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2431.207604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2431.318100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2431.380625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2431.490160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2431.574275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2432.487816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_dropped -r 2 [ 2432.551535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 2 [ 2432.631795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_dropped -r 2 [ 2432.700240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 2 [ 2432.779633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_dropped -r 2 [ 2432.858057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 2 [ 2432.926683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_dropped -r 2 [ 2432.993383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 2 [ 2433.070915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 2433.109227] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 2433.516863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2433.582924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2433.650762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2433.725070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2433.802935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2433.852458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2433.920889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2433.987764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2434.055247] LNet: 88735:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2434.101683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2434.147480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2434.210938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2434.271431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2434.318909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2434.399894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2435.467468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_aborted -r 2 [ 2435.554659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 2 [ 2435.645343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_aborted -r 2 [ 2435.723553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 2 [ 2435.808755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_aborted -r 2 [ 2435.901804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 2 [ 2435.991746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_aborted -r 2 [ 2436.063676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 2 [ 2436.116511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 2436.152182] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 2436.629430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2436.714345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2436.778882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2436.834927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2436.916914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2437.023467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2437.108819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2437.169724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2437.211339] LNet: 88856:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2437.269816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2437.352421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2437.428043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2437.501913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2437.578199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2437.688786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2438.582065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_no_route -r 2 [ 2438.629806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 2 [ 2438.676790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_no_route -r 2 [ 2438.731899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 2 [ 2438.787125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_no_route -r 2 [ 2438.833723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 2 [ 2438.889860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_no_route -r 2 [ 2438.974597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 2 [ 2439.037400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 2439.065137] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 2439.433253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2439.486415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2439.540668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2439.602305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2439.650601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2439.689867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2439.739989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2439.775701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2439.807897] LNet: 88976:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2439.830931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2439.902484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2439.960289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2440.016308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2440.075761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2440.136094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2440.882263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_timeout -r 2 [ 2440.918820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 2 [ 2440.968971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_timeout -r 2 [ 2441.024947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 2 [ 2441.090541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_timeout -r 2 [ 2441.152054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 2 [ 2441.239451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_timeout -r 2 [ 2441.305571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 2 [ 2441.373373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 2441.804803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2441.839337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2441.874973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2441.906950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2441.944915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2441.981828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2442.005416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2442.047922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2442.078420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2442.115566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2442.146486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2442.172915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2442.204966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2442.240335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2442.844950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_dropped -r 2 [ 2442.896855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 2 [ 2442.943602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_dropped -r 2 [ 2442.993573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 2 [ 2443.065586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_dropped -r 2 [ 2443.132854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 2 [ 2443.204302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_dropped -r 2 [ 2443.264612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 2 [ 2443.329762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 2443.354419] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 2443.358548] LNet: Skipped 1 previous similar message [ 2443.360748] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 2443.653774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2443.713410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2443.761576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2443.815150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2443.861518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2443.894598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2443.914712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2443.937902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2443.949783] LNet: 89216:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2443.953782] LNet: 89216:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 2443.966313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2443.988610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2444.024077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2444.057548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2444.088203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2444.123672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2452.672868] LNet: 89296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2452.685159] LNetError: 89296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2452.694867] LNet: Removed LNI 192.168.204.42@tcp [ 2452.725184] LNet: Removed LNI 192.168.204.42@tcp1 [ 2453.392658] Key type .llcrypt unregistered [ 2453.394180] Key type ._llcrypt unregistered [ 2478.849639] Key type ._llcrypt registered [ 2478.851191] Key type .llcrypt registered [ 2480.756850] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 13:16:15 (1786036575) [ 2481.481213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2481.526478] alg: No test for adler32 (adler32-zlib) [ 2482.626542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2482.760948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2482.832563] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2484.231692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2484.280595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.42@tcp [ 2484.313959] LNetError: 90278:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 2484.432718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.42@tcp [ 2484.567981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2494.136765] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 13:16:29 (1786036589) [ 2494.437597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2494.496270] LNet: 90867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2494.519755] LNetError: 90867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2494.538655] LNet: Removed LNI 192.168.204.42@tcp [ 2494.564450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2494.632453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2494.659342] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2495.559316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2495.600738] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 2496.547017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.42@tcp [ 2496.580929] LNetError: 91134:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 2496.672941] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 2496.676821] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 2497.987926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.42@tcp [ 2507.925584] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 13:16:43 (1786036603) [ 2510.468236] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg442-server [ 2512.996961] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 13:16:48 (1786036608) [ 2516.196853] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg442-server [ 2519.083084] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 13:16:53 (1786036613) [ 2522.206445] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg442-server [ 2524.186201] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 13:17:00 (1786036620) [ 2524.635711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2524.687690] LNet: 92756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2524.701949] LNetError: 92756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2524.705552] LNet: Removed LNI 192.168.204.42@tcp [ 2524.748624] LNet: Removed LNI 192.168.204.42@tcp1 [ 2524.821213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2524.899354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2524.947329] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2525.785587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2525.836979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2525.880568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2526.002962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2526.089926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2526.130389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2536.050293] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 13:17:11 (1786036631) [ 2536.498988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2536.527498] LNet: 93481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2536.535181] LNetError: 93481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2536.543727] LNetError: 93481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2536.564056] LNet: Removed LNI 192.168.204.42@tcp [ 2536.588742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2536.648980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2536.676967] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2545.769885] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 13:17:21 (1786036641) [ 2546.087865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2546.117515] LNet: 94185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2546.132632] LNetError: 94185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2546.151042] LNet: Removed LNI 192.168.204.42@tcp [ 2546.179307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2546.241280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2546.266024] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2556.040444] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 13:17:30 (1786036650) [ 2558.722337] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2560.967621] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 13:17:36 (1786036656) [ 2561.707219] LNet: 95222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2561.717149] LNetError: 95222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2561.732560] LNet: Removed LNI 192.168.204.42@tcp [ 2562.620959] Key type .llcrypt unregistered [ 2562.625430] Key type ._llcrypt unregistered [ 2571.377325] Key type ._llcrypt registered [ 2571.387710] Key type .llcrypt registered [ 2571.617015] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2571.640199] alg: No test for adler32 (adler32-zlib) [ 2572.683690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2572.720739] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2582.198762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 2588.305638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.142@tcp [ 2589.046929] LNet: 95650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2589.052848] LNetError: 95650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2589.077157] LNet: Removed LNI 192.168.204.42@tcp [ 2590.034659] Key type .llcrypt unregistered [ 2590.046137] Key type ._llcrypt unregistered [ 2608.876572] Key type ._llcrypt registered [ 2608.879352] Key type .llcrypt registered [ 2610.832593] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2613.929218] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2618.174443] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 13:18:32 (1786036712) [ 2621.771146] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2627.100837] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 13:18:40 (1786036720) [ 2632.549279] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg442-server [ 2635.537988] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 13:18:50 (1786036730) [ 2639.550910] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg442-server [ 2642.862726] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 13:18:57 (1786036737) [ 2646.636217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2646.728316] alg: No test for adler32 (adler32-zlib) [ 2647.914738] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 2648.318752] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2650.119514] Key type lgssc registered [ 2652.301907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2688.695806] Key type lgssc unregistered [ 2689.104836] LNet: 98836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2689.114520] LNetError: 98836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2689.136352] LNet: Removed LNI 192.168.204.42@tcp [ 2690.368389] Key type .llcrypt unregistered [ 2690.369850] Key type ._llcrypt unregistered [ 2716.888228] Key type ._llcrypt registered [ 2716.898448] Key type .llcrypt registered [ 2718.858231] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 13:20:13 (1786036813) [ 2719.720435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2719.760583] alg: No test for adler32 (adler32-zlib) [ 2720.943895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2721.203459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2721.321664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2721.454976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2721.558434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2733.502710] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 13:20:27 (1786036827) [ 2733.771732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2733.864848] LNet: 100251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2733.909763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2734.022670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2734.075397] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2735.015565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2735.028906] LNet: 100398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2735.033486] LNetError: 100398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2735.044661] LNet: Removed LNI 192.168.204.42@tcp [ 2735.067325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2735.114882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2735.144861] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2735.567654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2735.588607] LNetError: 100426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2735.620996] LNet: Removed LNI 192.168.204.42@tcp [ 2735.681866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2735.764313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2735.810706] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2746.853402] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 13:20:40 (1786036840) [ 2747.083166] LNet: 101009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2747.102381] LNet: 101009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2747.122722] LNetError: 101009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2747.167966] LNet: Removed LNI 192.168.204.42@tcp [ 2748.461828] Key type .llcrypt unregistered [ 2748.472835] Key type ._llcrypt unregistered [ 2758.693401] Key type ._llcrypt registered [ 2758.701654] Key type .llcrypt registered [ 2759.261730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2759.316586] alg: No test for adler32 (adler32-zlib) [ 2760.440133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2760.515594] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2760.518294] LNet: *** cfs_fail_loc=e003, val=100*** [ 2761.277375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2761.298664] LNet: *** cfs_fail_loc=e003, val=100*** [ 2761.300805] LNet: Skipped 2 previous similar messages [ 2761.311476] Ping reply received with mapping: [ 2761.314595] found metadata reply NI[192.168.204.42@tcp] [ 2761.317339] 2a cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2761.331931] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2761.340246] 5a 5a 5a 5a 5a 5a 5a 5a [ 2761.545457] LNet: 101420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2761.574613] LNetError: 101420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2761.633801] LNet: Removed LNI 192.168.204.42@tcp [ 2762.955308] Key type .llcrypt unregistered [ 2762.963544] Key type ._llcrypt unregistered [ 2780.172842] Key type ._llcrypt registered [ 2780.175050] Key type .llcrypt registered [ 2782.442653] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 13:21:17 (1786036877) [ 2783.748426] Key type .llcrypt unregistered [ 2783.751268] Key type ._llcrypt unregistered [ 2795.036576] Key type ._llcrypt registered [ 2795.038135] Key type .llcrypt registered [ 2795.411758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2795.448296] alg: No test for adler32 (adler32-zlib) [ 2797.300731] LNetError: 102482:0:(api-ni.c:2849:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2798.304301] Key type .llcrypt unregistered [ 2798.310519] Key type ._llcrypt unregistered [ 2811.729398] Key type ._llcrypt registered [ 2811.751655] Key type .llcrypt registered [ 2814.339909] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 13:21:48 (1786036908) [ 2815.213271] Key type .llcrypt unregistered [ 2815.220470] Key type ._llcrypt unregistered [ 2827.652840] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2827.664164] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2830.733149] Key type ._llcrypt registered [ 2830.740940] Key type .llcrypt registered [ 2831.094444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2831.142369] alg: No test for adler32 (adler32-zlib) [ 2832.288994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2832.400043] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2845.052638] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 13:22:20 (1786036940) [ 2845.460770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2845.538433] LNet: 104276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2845.562515] LNetError: 104276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2845.593839] LNet: Removed LNI 10.1.2.3@tcp [ 2845.643284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2847.532867] Key type .llcrypt unregistered [ 2847.539064] Key type ._llcrypt unregistered [ 2857.774972] Key type ._llcrypt registered [ 2857.776258] Key type .llcrypt registered [ 2858.083202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2858.106423] alg: No test for adler32 (adler32-zlib) [ 2859.440034] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2859.721920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2860.149326] LNet: 104570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2860.163262] LNetError: 104570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2860.189872] LNet: Removed LNI 192.168.204.42@tcp [ 2862.070762] Key type .llcrypt unregistered [ 2862.077702] Key type ._llcrypt unregistered [ 2883.933500] Key type ._llcrypt registered [ 2883.937597] Key type .llcrypt registered [ 2887.220112] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 13:23:00 (1786036980) [ 2888.545471] Key type .llcrypt unregistered [ 2888.549556] Key type ._llcrypt unregistered [ 2901.164781] Key type ._llcrypt registered [ 2901.167823] Key type .llcrypt registered [ 2901.548404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2901.590589] alg: No test for adler32 (adler32-zlib) [ 2911.759571] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 13:23:26 (1786037006) [ 2912.092877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2912.200495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2912.390509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2912.434126] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2913.699581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2913.763668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2913.836060] LNet: Removed LNI 192.168.204.42@tcp [ 2914.377344] LNet: 106332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2915.264462] Key type .llcrypt unregistered [ 2915.266224] Key type ._llcrypt unregistered [ 2935.855768] Key type ._llcrypt registered [ 2935.859255] Key type .llcrypt registered [ 2937.919321] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 13:23:52 (1786037032) [ 2938.659425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2938.684409] alg: No test for adler32 (adler32-zlib) [ 2939.736021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2939.910593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2939.998377] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2941.055740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2941.106791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.42@tcp [ 2941.147796] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2941.155132] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2941.166370] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2941, 0, 0 [ 2941.179877] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2941.184571] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2941.194767] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2941.203538] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2941, 0, 0 [ 2941.213631] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2941.218809] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2941.230390] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2941.234671] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2941, 0, 2991 [ 2941.240381] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2941.243773] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2941.247707] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2941.251414] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2941, 0, 2991 [ 2941.259210] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2951.452639] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 13:24:06 (1786037046) [ 2952.534154] LNet: 107874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2952.557037] LNetError: 107874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2952.598721] LNet: Removed LNI 192.168.204.42@tcp [ 2953.768699] Key type .llcrypt unregistered [ 2953.771974] Key type ._llcrypt unregistered [ 2964.579274] Key type ._llcrypt registered [ 2964.586782] Key type .llcrypt registered [ 2964.943561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2964.989557] alg: No test for adler32 (adler32-zlib) [ 2966.171153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2966.247211] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2979.063829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 2989.741804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2989.839102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2989.879475] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 2991.353564] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2991.480554] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2991.489170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2991.497544] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2993.907659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2993.928547] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 3007.332368] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 13:25:03 (1786037103) [ 3007.708416] LNet: 109405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3007.718982] LNetError: 109405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3007.732533] LNet: Removed LNI 192.168.204.42@tcp [ 3008.799948] LNet: Removed LNI 192.168.204.42@tcp1 [ 3008.801933] LNetError: 109405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3008.820454] LNetError: 109405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3008.852752] LNet: Removed LNI 10.1.2.3@tcp99 [ 3009.456489] Key type .llcrypt unregistered [ 3009.459215] Key type ._llcrypt unregistered [ 3020.282876] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3020.295850] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3023.134099] Key type ._llcrypt registered [ 3023.137375] Key type .llcrypt registered [ 3023.437792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3023.447608] alg: No test for adler32 (adler32-zlib) [ 3024.505376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3024.588594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3024.616208] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3025.187440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3025.231039] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3027.942733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 3028.020781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3028.086789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 3028.123658] LNetError: 110154:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 3028.185226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3028.249993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3028.896145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3029.033789] LNet: 110174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3029.049774] LNetError: 110174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3029.057110] LNet: Removed LNI 10.1.2.3@tcp [ 3030.122609] LNet: Removed LNI 192.168.204.42@tcp [ 3031.130458] Key type .llcrypt unregistered [ 3031.131810] Key type ._llcrypt unregistered [ 3047.633948] Key type ._llcrypt registered [ 3047.639109] Key type .llcrypt registered [ 3049.789026] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 13:25:44 (1786037144) [ 3050.413494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3050.447303] alg: No test for adler32 (adler32-zlib) [ 3051.417001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3051.505760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3051.546648] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3052.087541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 3052.156976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg442-client@tcp [ 3059.887680] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 13:25:55 (1786037155) [ 3060.137677] LNet: 111712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3060.150683] LNetError: 111712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3060.186173] LNet: Removed LNI 192.168.204.42@tcp [ 3061.185505] Key type .llcrypt unregistered [ 3061.188929] Key type ._llcrypt unregistered [ 3068.988040] Key type ._llcrypt registered [ 3068.992393] Key type .llcrypt registered [ 3069.241665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3069.295966] alg: No test for adler32 (adler32-zlib) [ 3070.334972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 3070.372081] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3070.416746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 3078.423774] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 13:26:13 (1786037173) [ 3080.466486] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 3082.798591] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 13:26:18 (1786037178) [ 3084.312687] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 3086.289478] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 13:26:21 (1786037181) [ 3086.513213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3086.569929] LNet: 113134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3086.579606] LNetError: 113134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3086.646656] LNet: Removed LNI 192.168.204.42@tcp [ 3086.698211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3086.835961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3086.884182] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3088.300580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 3088.419580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 3098.316795] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 13:26:33 (1786037193) [ 3098.437920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3098.476056] LNet: 113841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3098.493131] LNetError: 113841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3099.560171] LNet: Removed LNI 192.168.204.42@tcp [ 3099.601233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3099.714503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 3099.783821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3100.459615] LNet: 113887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3101.085373] Key type .llcrypt unregistered [ 3101.091186] Key type ._llcrypt unregistered [ 3119.058451] Key type ._llcrypt registered [ 3119.062653] Key type .llcrypt registered [ 3120.858898] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 13:26:56 (1786037216) [ 3122.030244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3122.072626] alg: No test for adler32 (adler32-zlib) [ 3123.231326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3123.340896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3123.432598] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3124.111786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3124.157988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3133.452988] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 13:27:08 (1786037228) [ 3133.706710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3133.764722] LNet: 115409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3133.769811] LNetError: 115409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3133.799352] LNet: Removed LNI 192.168.204.42@tcp [ 3133.843464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3133.947259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 3134.027464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 3142.760816] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 13:27:18 (1786037238) [ 3142.866612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3142.921221] LNet: 115985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3142.963774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3143.082268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3143.760716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3143.828175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3143.938762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 3144.574362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3144.628783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3144.761499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3144.804586] LNet: 116024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3144.866791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3145.007943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3145.412630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3145.470321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3145.574515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 3146.245814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3146.308196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3154.748522] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 13:27:30 (1786037250) [ 3154.951311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3154.987295] LNet: 116623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3155.028562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3155.134961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3155.174744] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3156.283450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3156.307873] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 3157.428623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3157.510700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3157.880295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3157.944552] LNetError: 116904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3157.956712] LNet: Removed LNI 192.168.204.42@tcp [ 3157.991413] LNet: Removed LNI 192.168.204.42@tcp2 [ 3158.035874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3158.159339] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3158.186144] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 3158.200531] LNet: 116910:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3159.204924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3159.548340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3159.570793] LNet: 116948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3159.577251] LNet: 116948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 3159.599690] LNetError: 116948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3159.608079] LNetError: 116948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3159.624664] LNet: Removed LNI 192.168.204.42@tcp [ 3159.648787] LNet: Removed LNI 192.168.204.42@tcp2 [ 3159.680226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3159.769244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3159.802248] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3160.708443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3160.742768] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 3161.783181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3161.847463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 3161.889926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3162.047586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3162.062871] LNetError: 117232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3162.068755] LNetError: 117232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3162.074519] LNet: Removed LNI 192.168.204.42@tcp [ 3163.109242] LNet: Removed LNI 192.168.204.42@tcp2 [ 3163.181402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3163.307880] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3163.317395] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 3163.329304] LNet: 117238:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3164.394355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3164.736919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3164.785498] LNet: Removed LNI 192.168.204.42@tcp [ 3165.864339] LNet: Removed LNI 192.168.204.42@tcp2 [ 3165.921871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3165.996675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3166.043082] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3166.891026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3166.926393] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 3168.282581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3168.359457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3168.609547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3168.635113] LNet: 117555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3168.641272] LNet: 117555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 3168.650737] LNetError: 117555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3168.657602] LNetError: 117555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 3168.662481] LNet: Removed LNI 192.168.204.42@tcp [ 3168.673846] LNet: Removed LNI 192.168.204.42@tcp2 [ 3168.738659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3168.845234] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3168.850197] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 3168.861624] LNet: 117561:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3169.345554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3169.653417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3170.785224] LNet: Removed LNI 192.168.204.42@tcp [ 3170.812250] LNet: Removed LNI 192.168.204.42@tcp2 [ 3170.908593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3170.989323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3171.035408] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3172.332529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3172.382496] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 3174.138805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3174.264114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 3174.411800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3174.777798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3175.903354] LNet: Removed LNI 192.168.204.42@tcp [ 3176.934280] LNet: Removed LNI 192.168.204.42@tcp2 [ 3177.017954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3177.152484] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3177.167438] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 3177.182590] LNet: 117889:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3178.068560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3178.280464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3178.341766] LNetError: 117926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3178.353739] LNetError: 117926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 3178.365631] LNet: Removed LNI 192.168.204.42@tcp [ 3179.446772] LNet: Removed LNI 192.168.204.42@tcp2 [ 3179.534717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3179.674020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3179.707213] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3180.715345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3180.754618] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 3181.802634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3181.835323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3181.889681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3182.334158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3183.391225] LNet: Removed LNI 192.168.204.42@tcp [ 3183.405171] LNet: Removed LNI 192.168.204.42@tcp2 [ 3183.493793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3183.576785] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3183.582633] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 3183.587146] LNet: 118230:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3183.982841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3184.588333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3185.631761] LNet: Removed LNI 192.168.204.42@tcp [ 3185.643805] LNet: Removed LNI 192.168.204.42@tcp2 [ 3185.719862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3185.954407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3186.035142] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3187.384232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3187.446109] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 3189.332933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3189.500544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 3189.594559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 3189.719470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3190.497472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3190.534866] LNet: 118584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3190.548509] LNet: 118584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 3191.583219] LNet: Removed LNI 192.168.204.42@tcp [ 3192.643343] LNet: Removed LNI 192.168.204.42@tcp2 [ 3192.731836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3193.041690] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3193.068174] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 3193.074749] LNet: 118590:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3194.524952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3203.846753] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 3206.080479] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 13:28:21 (1786037301) [ 3206.340509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3206.380511] LNetError: 119348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3206.390715] LNetError: 119348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 3206.403500] LNet: Removed LNI 192.168.204.42@tcp [ 3206.434197] LNet: Removed LNI 192.168.204.42@tcp2 [ 3206.510342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3206.901926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3206.943753] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3207.951369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3207.977856] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3209.287248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3209.313872] LNet: Removed LNI 10.1.2.3@tcp [ 3209.333941] LNet: Removed LNI 192.168.204.42@tcp [ 3209.384623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3209.529633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 3209.595469] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3209.605352] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3210.412063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 3210.460460] LNet: Removed LNI 192.168.204.42@tcp [ 3210.477559] LNet: Removed LNI 10.1.2.3@tcp [ 3218.874794] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 13:28:34 (1786037314) [ 3219.213640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3219.264377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3219.340638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3219.379530] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3220.376306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 3220.460374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 3220.510736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 3220.543597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 3220.568435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 3220.603605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 3220.645333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 3220.675320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 3220.713511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 3220.764279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 3220.811920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 3221.036755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 3228.888902] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 13:28:44 (1786037324) [ 3229.126767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3229.147673] LNet: 120962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3229.155228] LNet: 120962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 3230.189499] LNet: Removed LNI 192.168.204.42@tcp [ 3230.265811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3230.527024] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 3230.808399] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3230.822612] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3230.833394] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 3233.400353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3233.460602] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3235.079367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3235.125307] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3238.370059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3238.426391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 3238.527359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.42@tcp -m PUT -e local_timeout -r 1 [ 3238.640254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 3238.748144] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3238.752658] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3239.188159] LNetError: 121027:0:(lib-move.c:4801:lnet_parse()) 192.168.204.42@tcp, src 192.168.204.42@tcp: Dropping PUT (error -108 looking up sender) [ 3240.226243] LNetError: 121490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3240.228788] LNetError: 121490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 3240.240588] LNet: Removed LNI 10.1.2.3@tcp [ 3241.257436] LNet: Removed LNI 192.168.204.42@tcp [ 3241.886444] Key type .llcrypt unregistered [ 3241.888850] Key type ._llcrypt unregistered [ 3262.795267] Key type ._llcrypt registered [ 3262.801852] Key type .llcrypt registered [ 3264.718816] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 13:29:19 (1786037359) [ 3265.372414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3265.405990] alg: No test for adler32 (adler32-zlib) [ 3266.519542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3266.982807] LNet: 122344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3267.803366] Key type .llcrypt unregistered [ 3267.804819] Key type ._llcrypt unregistered [ 3276.584563] Key type ._llcrypt registered [ 3276.592417] Key type .llcrypt registered [ 3276.935876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3276.968955] alg: No test for adler32 (adler32-zlib) [ 3277.997315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3278.046072] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3287.929054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 3288.230227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 3288.993789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.142@tcp [ 3289.248352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3289.311230] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 3289.319228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3289.385137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3289.437790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3289.491986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3289.533656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3289.574953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3289.606258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3289.625464] LNet: 122798:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3289.641916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3289.668778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3289.717443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3289.778812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3289.826406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3289.870147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3290.820514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3291.041643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3291.069115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3291.093876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3291.117802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3291.145884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3291.170710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3291.196498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3291.223098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3291.240813] LNet: 122916:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3291.262290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3291.289474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3291.345761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3291.391176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3291.431893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3291.463112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3292.265943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.142@tcp [ 3292.566703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3292.592640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3292.625216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3292.654760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3292.700886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3292.740480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3292.843858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3292.932925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3292.988167] LNet: 123033:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3293.041707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3293.084755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3293.109757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3293.134701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3293.176378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3293.210932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3293.895078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3294.284645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3294.326470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3294.362543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3294.390808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3294.435874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3294.501892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3294.544842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3294.585443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3294.627431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3294.681170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3294.730445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3294.773951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3294.808880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3294.844172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3295.549862] LNet: 123200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3295.559979] LNetError: 123200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3295.595838] LNet: Removed LNI 192.168.204.42@tcp [ 3296.574185] Key type .llcrypt unregistered [ 3296.576360] Key type ._llcrypt unregistered [ 3323.536203] Key type ._llcrypt registered [ 3323.542154] Key type .llcrypt registered [ 3325.513215] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 13:30:20 (1786037420) [ 3326.251480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3326.293304] alg: No test for adler32 (adler32-zlib) [ 3327.319569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3327.618353] LNet: 124089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3328.292358] Key type .llcrypt unregistered [ 3328.305538] Key type ._llcrypt unregistered [ 3340.102974] Key type ._llcrypt registered [ 3340.112303] Key type .llcrypt registered [ 3340.616273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3340.687797] alg: No test for adler32 (adler32-zlib) [ 3341.733796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3341.779812] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3352.087901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 3365.027036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3365.103149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3365.122496] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 3366.908965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3367.930173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.142@tcp [ 3368.428362] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 3368.470226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3368.572785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3368.631976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3368.673840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3368.732607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3368.791280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3368.840907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3368.897753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3368.942244] LNet: 124747:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3368.990326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3369.061255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3369.109661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3369.162533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3369.229596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3369.326294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3370.365504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3370.832445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3370.872586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3370.928565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3370.979550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3371.040727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3371.085827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3371.132902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3371.169602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3371.199535] LNet: 124864:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3371.228827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3371.289649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3371.324712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3371.376843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3371.424627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3371.457379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3372.456817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.142@tcp [ 3372.933327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3373.003403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3373.057978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3373.093636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3373.143783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3373.193860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3373.240445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3373.304943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3373.349309] LNet: 124982:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3373.408280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3373.471961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3373.527623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3373.570691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3373.645656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3373.726880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3374.979442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3375.325422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3375.372801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3375.444849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3375.500541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3375.536707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3375.576257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3375.643421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3375.706433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3375.757477] LNet: 125099:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3375.782578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3375.842264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3375.880223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3375.917596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3375.951613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3376.014955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3382.321297] LNet: 125184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3382.335561] LNetError: 125184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3382.342447] LNet: Removed LNI 192.168.204.42@tcp [ 3382.380932] LNet: Removed LNI 192.168.204.42@tcp1 [ 3383.091692] Key type .llcrypt unregistered [ 3383.101779] Key type ._llcrypt unregistered [ 3413.399324] Key type ._llcrypt registered [ 3413.406164] Key type .llcrypt registered [ 3416.419510] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 13:31:49 (1786037509) [ 3418.101810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3418.201099] alg: No test for adler32 (adler32-zlib) [ 3419.515109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3419.570130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3419.589247] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3420.447489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3420.487695] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 3422.197910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 3422.291457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 3423.327176] LNet: Removed LNI 192.168.204.42@tcp [ 3433.319355] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 13:32:08 (1786037528) [ 3434.984168] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 3437.720071] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 13:32:12 (1786037532) [ 3449.529835] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 13:32:24 (1786037544) [ 3465.856611] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 13:32:39 (1786037559) [ 3480.285242] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 13:32:54 (1786037574) [ 3491.801295] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 13:33:06 (1786037586) [ 3501.695704] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 13:33:16 (1786037596) [ 3512.071832] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 13:33:27 (1786037607) [ 3522.846908] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 13:33:38 (1786037618) [ 3533.374152] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 13:33:48 (1786037628) [ 3545.394936] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 13:34:00 (1786037640) [ 3557.483418] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 13:34:11 (1786037651) [ 3570.457188] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 13:34:24 (1786037664) [ 3582.815056] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 13:34:38 (1786037678) [ 3594.343847] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 13:34:49 (1786037689) [ 3608.796508] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 13:35:04 (1786037704) [ 3619.736400] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 13:35:15 (1786037715) [ 3629.428677] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 13:35:24 (1786037724) [ 3640.247477] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 13:35:36 (1786037736) [ 3640.964587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3641.004580] LNet: 136895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3641.017726] LNetError: 136895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3641.050135] LNet: Removed LNI 192.168.204.42@tcp2 [ 3641.126304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3641.321433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3641.365107] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3642.849719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3642.892895] LNet: 137037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3642.908981] LNetError: 137037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3642.956881] LNet: Removed LNI 192.168.204.42@tcp [ 3643.013549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3643.139130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 3643.184864] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3653.846303] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 13:35:48 (1786037748) [ 3667.968080] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 13:36:02 (1786037762) [ 3679.622335] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 13:36:15 (1786037775) [ 3689.306968] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 13:36:24 (1786037784) [ 3698.960879] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 13:36:34 (1786037794) [ 3712.326826] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 13:36:47 (1786037807) [ 3723.696401] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 13:36:58 (1786037818) [ 3734.832728] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 13:37:09 (1786037829) [ 3745.202836] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 13:37:20 (1786037840) [ 3756.182396] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 13:37:31 (1786037851) [ 3764.787452] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 13:37:40 (1786037860) [ 3773.204956] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 13:37:48 (1786037868) [ 3784.597276] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 3529 sec ========== 13:37:59 (1786037879) [ 3785.279484] LNet: 144510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3785.301846] LNetError: 144510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3785.323887] LNet: Removed LNI 192.168.204.42@tcp [ 3786.346565] Key type .llcrypt unregistered [ 3786.349060] Key type ._llcrypt unregistered [ 3806.848509] Key type ._llcrypt registered [ 3806.860588] Key type .llcrypt registered [ 3807.599915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3807.626395] alg: No test for adler32 (adler32-zlib) [ 3808.902750] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 3809.324351] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 3811.024071] Key type lgssc registered [ 3813.477594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3896.153491] Lustre: Mounted lustre-client - version 2.17.56_53_g253671b [ 3904.854059] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3914.462585] Lustre: Unmounted lustre-client [ 3949.793832] Key type lgssc unregistered [ 3950.305464] LNet: 147477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3950.337261] LNetError: 147477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3950.353301] LNet: Removed LNI 192.168.204.42@tcp [ 3951.475147] Key type .llcrypt unregistered [ 3951.476787] Key type ._llcrypt unregistered