[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 443736006 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003317] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008316] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010027] pid_max: default: 32768 minimum: 301 [ 0.011180] LSM: Security Framework initializing [ 0.012090] Yama: becoming mindful. [ 0.013049] SELinux: Initializing. [ 0.014110] *** VALIDATE selinux *** [ 0.024407] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029265] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032122] *** VALIDATE tmpfs *** [ 0.034384] *** VALIDATE proc *** [ 0.035260] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.037260] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040044] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.045546] debug: unmapping init [mem 0xffffffff9fe59000-0xffffffff9fe60fff] [ 0.047191] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048683] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049033] ... version: 2 [ 0.050015] ... bit width: 48 [ 0.051014] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054019] ... fixed-purpose events: 3 [ 0.055014] ... event mask: 000000070000000f [ 0.056362] rcu: Hierarchical SRCU implementation. [ 0.058483] smp: Bringing up secondary CPUs ... [ 0.059619] x86: Booting SMP configuration: [ 0.060027] .... node #0, CPUs: #1 #2 #3 [ 0.064013] smp: Brought up 1 node, 4 CPUs [ 0.066019] smpboot: Max logical packages: 1 [ 0.067015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149020] node 0 deferred pages initialised in 80ms [ 0.152151] devtmpfs: initialized [ 0.153199] x86/mm: Memory block size: 128MB [ 0.155697] gcov: version magic: 0x41383552 [ 0.156635] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162342] pinctrl core: initialized pinctrl subsystem [ 0.164151] [ 0.164694] ************************************************************* [ 0.166012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168012] ** ** [ 0.170012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172012] ** ** [ 0.174012] ** This means that this kernel is built to expose internal ** [ 0.176014] ** IOMMU data structures, which may compromise security on ** [ 0.179012] ** your system. ** [ 0.181011] ** ** [ 0.183013] ** If you see this message and you are not debugging the ** [ 0.185022] ** kernel, report this immediately to your vendor! ** [ 0.187012] ** ** [ 0.189012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191011] ************************************************************* [ 0.192600] NET: Registered protocol family 16 [ 0.194410] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.204012] cpuidle: using governor menu [ 0.205842] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.207462] PCI: Using configuration type 1 for base access [ 0.209094] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218108] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225108] cryptd: max_cpu_qlen set to 1000 [ 0.229339] ACPI: Added _OSI(Module Device) [ 0.231017] ACPI: Added _OSI(Processor Device) [ 0.233014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.235025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.239174] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.244476] ACPI: Interpreter enabled [ 0.246061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.247012] ACPI: Using IOAPIC for interrupt routing [ 0.249093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252408] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.262807] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.265038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.268021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.271080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.276394] acpiphp: Slot [2] registered [ 0.278106] acpiphp: Slot [5] registered [ 0.279112] acpiphp: Slot [6] registered [ 0.281114] acpiphp: Slot [3] registered [ 0.282078] acpiphp: Slot [4] registered [ 0.283079] acpiphp: Slot [7] registered [ 0.285086] acpiphp: Slot [8] registered [ 0.286095] acpiphp: Slot [9] registered [ 0.287082] acpiphp: Slot [10] registered [ 0.288075] acpiphp: Slot [11] registered [ 0.289096] acpiphp: Slot [12] registered [ 0.290072] acpiphp: Slot [13] registered [ 0.292092] acpiphp: Slot [14] registered [ 0.293000] acpiphp: Slot [15] registered [ 0.293000] acpiphp: Slot [16] registered [ 0.294077] acpiphp: Slot [17] registered [ 0.296077] acpiphp: Slot [18] registered [ 0.297081] acpiphp: Slot [19] registered [ 0.298092] acpiphp: Slot [20] registered [ 0.299067] acpiphp: Slot [21] registered [ 0.299985] acpiphp: Slot [22] registered [ 0.301113] acpiphp: Slot [23] registered [ 0.302071] acpiphp: Slot [24] registered [ 0.303078] acpiphp: Slot [25] registered [ 0.304089] acpiphp: Slot [26] registered [ 0.305077] acpiphp: Slot [27] registered [ 0.307157] acpiphp: Slot [28] registered [ 0.308129] acpiphp: Slot [29] registered [ 0.310138] acpiphp: Slot [30] registered [ 0.311117] acpiphp: Slot [31] registered [ 0.313121] PCI host bridge to bus 0000:00 [ 0.314023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.317029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.320031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.323026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.325043] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.328030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.330219] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.335934] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.342456] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.350016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.354036] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.358018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.360016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362691] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.365843] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.369048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372844] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.375959] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.387693] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.392022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.398289] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.404015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.408021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.419951] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.427944] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.436016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.443015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.463016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.475258] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.477478] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.479275] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.481463] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.484236] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.488096] iommu: Default domain type: Passthrough [ 0.490345] SCSI subsystem initialized [ 0.492138] ACPI: bus type USB registered [ 0.494118] usbcore: registered new interface driver usbfs [ 0.495068] usbcore: registered new interface driver hub [ 0.497063] usbcore: registered new device driver usb [ 0.498151] pps_core: LinuxPPS API ver. 1 registered [ 0.499009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.502066] PTP clock support registered [ 0.504052] EDAC MC: Ver: 3.0.0 [ 0.505103] PCI: Using ACPI for IRQ routing [ 0.507939] NetLabel: Initializing [ 0.509013] NetLabel: domain hash size = 128 [ 0.511013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.513094] NetLabel: unlabeled traffic allowed by default [ 0.516037] vgaarb: loaded [ 0.517257] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.518016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.524005] clocksource: Switched to clocksource kvm-clock [ 0.630470] VFS: Disk quotas dquot_6.6.0 [ 0.632239] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.634243] *** VALIDATE ramfs *** [ 0.635931] *** VALIDATE hugetlbfs *** [ 0.638361] pnp: PnP ACPI init [ 0.640861] pnp: PnP ACPI: found 6 devices [ 0.669037] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.673076] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.675698] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.677992] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.680532] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.683143] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.686128] NET: Registered protocol family 2 [ 0.689376] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.693927] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.698740] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.704243] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.707732] TCP: Hash tables configured (established 65536 bind 65536) [ 0.710707] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.713467] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.716974] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.719935] NET: Registered protocol family 1 [ 0.723060] RPC: Registered named UNIX socket transport module. [ 0.725594] RPC: Registered udp transport module. [ 0.727535] RPC: Registered tcp transport module. [ 0.729486] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.731952] NET: Registered protocol family 44 [ 0.733728] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.736036] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.738244] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.740759] PCI: CLS 0 bytes, default 64 [ 0.742408] Unpacking initramfs... [ 2.156275] debug: unmapping init [mem 0xffff98fdbcc64000-0xffff98fdbffcffff] [ 2.160815] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.162760] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.165157] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.653458] Initialise system trusted keyrings [ 2.655204] Key type blacklist registered [ 2.657405] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.667667] zbud: loaded [ 2.670710] *** VALIDATE nfs *** [ 2.671939] *** VALIDATE nfs4 *** [ 2.673389] pstore: using deflate compression [ 2.678141] Platform Keyring initialized [ 2.774132] NET: Registered protocol family 38 [ 2.775927] Key type asymmetric registered [ 2.777332] Asymmetric key parser 'x509' registered [ 2.778924] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.781249] io scheduler mq-deadline registered [ 2.782534] io scheduler kyber registered [ 2.783799] io scheduler bfq registered [ 2.785577] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.788157] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.790495] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.793396] ACPI: Power Button [PWRF] [ 2.799099] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.805578] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.818295] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.847417] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.876505] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.881030] Non-volatile memory driver v1.3 [ 2.882458] Linux agpgart interface v0.103 [ 2.911693] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.913991] vda: detected capacity change from 0 to 75100160 [ 2.928299] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.931105] vdb: detected capacity change from 0 to 1073741824 [ 2.937364] libphy: Fixed MDIO Bus: probed [ 2.942415] usbcore: registered new interface driver usbserial_generic [ 2.944648] usbserial: USB Serial support registered for generic [ 2.947508] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.952689] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.954858] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.958371] mousedev: PS/2 mouse device common for all mice [ 2.963049] rtc_cmos 00:05: RTC can wake from S4 [ 2.967670] rtc_cmos 00:05: registered as rtc0 [ 2.969730] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.972318] intel_pstate: CPU model not supported [ 2.972390] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.977545] hid: raw HID events driver (C) Jiri Kosina [ 2.980175] usbcore: registered new interface driver usbhid [ 2.980456] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.983093] usbhid: USB HID core driver [ 2.988215] drop_monitor: Initializing network drop monitor service [ 2.989111] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.990776] Initializing XFRM netlink socket [ 2.996399] NET: Registered protocol family 10 [ 3.000478] Segment Routing with IPv6 [ 3.002079] NET: Registered protocol family 17 [ 3.004800] mpls_gso: MPLS GSO support [ 3.010527] RAS: Correctable Errors collector initialized. [ 3.012853] AVX version of gcm_enc/dec engaged. [ 3.015747] AES CTR mode by8 optimization enabled [ 3.083663] sched_clock: Marking stable (3083645103, 0)->(3972157971, -888512868) [ 3.087497] registered taskstats version 1 [ 3.089231] Loading compiled-in X.509 certificates [ 3.091137] zswap: loaded using pool lzo/zbud [ 3.113304] Key type big_key registered [ 3.123213] Key type encrypted registered [ 3.124992] ima: No TPM chip found, activating TPM-bypass! [ 3.126928] ima: Allocated hash algorithm: sha1 [ 3.128188] ima: No architecture policies found [ 3.129761] evm: Initialising EVM extended attributes: [ 3.131228] evm: security.selinux [ 3.132231] evm: security.ima [ 3.133146] evm: security.capability [ 3.134220] evm: HMAC attrs: 0x1 [ 3.136281] rtc_cmos 00:05: setting system clock to 2026-09-08 02:02:17 UTC (1788832937) [ 3.141812] debug: unmapping init [mem 0xffffffffa0e03000-0xffffffffa0ffffff] [ 3.144437] debug: unmapping init [mem 0xffffffff9fb82000-0xffffffff9fe58fff] [ 3.153436] Write protecting the kernel read-only data: 28672k [ 3.156654] debug: unmapping init [mem 0xffffffff9e203000-0xffffffff9e3fffff] [ 3.158688] debug: unmapping init [mem 0xffffffff9eb14000-0xffffffff9ebfffff] [ 3.189763] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.198565] systemd[1]: Detected virtualization kvm. [ 3.200833] systemd[1]: Detected architecture x86-64. [ 3.202726] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.229792] systemd[1]: No hostname configured. [ 3.231613] systemd[1]: Set hostname to . [ 3.233892] random: systemd: uninitialized urandom read (16 bytes read) [ 3.236400] systemd[1]: Initializing machine ID from random generator. [ 3.262944] random: ln: uninitialized urandom read (6 bytes read) [ 3.367588] random: systemd: uninitialized urandom read (16 bytes read) [ 3.370572] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.376142] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.380885] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.949693] device-mapper: uevent: version 1.0.3 [ 3.952138] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.582955] virtio_net virtio0 ens2: renamed from eth0 [ 4.627107] scsi host0: ata_piix [ 4.658757] scsi host1: ata_piix [ 4.660305] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.662668] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 5.675060] random: fast init done [ 8.828418] dracut-initqueue[590]: RTNETLINK answers: File exists [ 9.638739] random: crng init done [ 9.641831] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.856275] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.914199] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.161271] SELinux: Disabled at runtime. [ 11.229661] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.239978] systemd[1]: Detected virtualization kvm. [ 11.241969] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.702505] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.705977] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.711124] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.716499] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.719560] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.729569] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.735152] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [[ 11.854556] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.123452] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.418667] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.441238] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.498501] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.522975] EDAC sbridge: Ver: 1.1.2 [ 13.549382] Key type dns_resolver registered [ 13.856343] NFS: Registering the id_resolver key type [ 13.858455] Key type id_resolver registered [ 13.860104] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg412-client login: [ 73.193492] libcfs: loading out-of-tree module taints kernel. [ 73.418279] Key type ._llcrypt registered [ 73.419939] Key type .llcrypt registered [ 73.909359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 73.943815] alg: No test for adler32 (adler32-zlib) [ 75.595737] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 76.461401] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 78.207169] Key type lgssc registered [ 79.748305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 204.941109] Lustre: Mounted lustre-client - version 2.17.57_106_g45f4cc0 [ 207.125597] hrtimer: interrupt took 2619159 ns [ 209.638426] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 220.415742] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing check_logdir /tmp/testlogs/ [ 224.831121] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing yml_node [ 228.897526] Lustre: DEBUG MARKER: Client: 2.17.57.106 [ 230.374536] Lustre: lustre-OST0000-osc-ffff98fe06616800: disconnect after 23s idle [ 231.698484] Lustre: DEBUG MARKER: MDS: 2.17.57.106 [ 234.271956] Lustre: DEBUG MARKER: OSS: 2.17.57.106 [ 236.132726] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Sep 7 22:06:08 EDT 2026 [ 253.831735] Lustre: Unmounted lustre-client [ 288.912947] Key type lgssc unregistered [ 289.241423] LNet: 8404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 289.255827] LNetError: 8404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 289.274047] LNet: Removed LNI 192.168.204.12@tcp [ 290.114978] Key type .llcrypt unregistered [ 290.117132] Key type ._llcrypt unregistered [ 313.845504] Key type ._llcrypt registered [ 313.850107] Key type .llcrypt registered [ 314.057203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 314.082904] alg: No test for adler32 (adler32-zlib) [ 314.994418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 315.080888] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 315.097482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 319.230640] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 319.309290] LNet: 9275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.318143] LNetError: 9275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 319.342297] LNet: Removed LNI 192.168.204.12@tcp [ 320.017179] Key type .llcrypt unregistered [ 320.020890] Key type ._llcrypt unregistered [ 325.757529] Key type ._llcrypt registered [ 325.759994] Key type .llcrypt registered [ 327.018827] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 22:07:39 (1788833259) [ 327.550568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 327.570496] alg: No test for adler32 (adler32-zlib) [ 328.497951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.561436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.576298] LNet: 9690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.602593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.645666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 328.661369] LNet: 9697:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 334.923779] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 22:07:47 (1788833267) [ 335.030751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.066828] LNet: 10269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.098679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.235711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 341.123701] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 22:07:54 (1788833274) [ 341.210776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.247548] LNet: 10856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.274037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.379394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 346.922131] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 22:08:00 (1788833280) [ 347.019400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.046899] LNet: 11442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.073628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.170820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 352.995476] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 22:08:06 (1788833286) [ 353.188109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.214619] LNet: 12031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.250447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.337505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 353.771754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 359.822679] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 22:08:12 (1788833292) [ 360.027339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.084811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.195287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 360.556349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 361.025032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 367.585560] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 22:08:20 (1788833300) [ 367.741749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.775307] LNet: 13217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.782469] LNet: 13217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 367.809691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.885647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 374.312658] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 22:08:27 (1788833307) [ 374.410516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.447549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.568520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 374.608686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 374.939620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 374.997407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 375.254518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 375.279921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 375.559974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 375.587945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 375.862671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 375.892552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 376.271268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 376.304603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 376.593807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 376.620580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 376.944681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 376.981095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 377.305402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 377.362038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 377.687580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 377.715973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 378.077866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 378.135443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 378.441570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 378.481293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 378.760320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 378.803267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 384.486178] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 22:08:37 (1788833317) [ 384.583188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.604062] LNet: 14479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.613205] LNet: 14479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 384.659269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.758187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 384.800694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 391.455303] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 22:08:44 (1788833324) [ 391.665861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.736858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.796548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 391.831692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 397.902983] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 22:08:51 (1788833331) [ 398.039680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.070767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.131169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 398.153864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 404.301918] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 22:08:57 (1788833337) [ 404.589206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.646174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.742814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 404.792966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 411.433352] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 22:09:04 (1788833344) [ 411.654442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.741766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.813564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 411.891385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 417.771393] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 22:09:10 (1788833350) [ 417.879959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.902645] LNet: 17408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.911048] LNet: 17408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 417.947714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.036569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 418.073713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 425.150787] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 22:09:18 (1788833358) [ 425.245801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.313195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.417825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 425.791044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 432.058785] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 22:09:25 (1788833365) [ 432.162363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.198395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.405392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 432.652406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 439.099401] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 22:09:31 (1788833371) [ 439.233499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.295323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.477629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 439.710539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 445.720460] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 22:09:38 (1788833378) [ 445.915309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.956155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.479524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 446.761582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 453.591401] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 22:09:46 (1788833386) [ 453.776811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.846083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.922966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 453.987456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.983391] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 22:09:53 (1788833393) [ 461.208248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.252506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.326170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.369715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 467.932754] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 22:10:01 (1788833401) [ 467.987966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.027474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.122943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 474.725408] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 22:10:07 (1788833407) [ 474.820822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.870480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.997127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 480.956185] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 22:10:14 (1788833414) [ 481.138685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.217365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.346665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 481.957920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 488.888792] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 22:10:21 (1788833421) [ 489.057011] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.083371] LNet: 23440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.092786] LNet: 23440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 489.153317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.234643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 489.262574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 495.865036] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 22:10:28 (1788833428) [ 495.986147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.053193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.153453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 496.185488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 502.190146] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 22:10:35 (1788833435) [ 502.355451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.392263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.449543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 502.487724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 508.295260] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 22:10:41 (1788833441) [ 508.401048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.469078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.527733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 508.557203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 516.000569] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 22:10:48 (1788833448) [ 516.312787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.372790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.456121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 516.566446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 516.593991] LNetError: 25799:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 516.671020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 516.769349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 524.154361] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 22:10:57 (1788833457) [ 524.462523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.514512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.615311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.9RUhujB4ue [ 524.708872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.9RUhujB4ue [ 524.814063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.9RUhujB4ue [ 532.458515] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 22:11:04 (1788833464) [ 532.641512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.715125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.865436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 532.903553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 532.976743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 539.093497] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 22:11:12 (1788833472) [ 539.184364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.244586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.364322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 539.411099] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 541.240298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 542.140080] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 544.063268] Key type lgssc registered [ 545.145160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 548.669737] LNetError: 27558:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.12@tcp99 has no available nets [ 551.648379] Key type lgssc unregistered [ 552.126099] LNetError: 28665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 552.148931] LNet: Removed LNI 192.168.204.12@tcp [ 552.837284] Key type .llcrypt unregistered [ 552.839822] Key type ._llcrypt unregistered [ 559.037317] Key type ._llcrypt registered [ 559.042504] Key type .llcrypt registered [ 560.397721] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 22:11:33 (1788833493) [ 561.084862] Key type .llcrypt unregistered [ 561.087637] Key type ._llcrypt unregistered [ 568.993447] Key type ._llcrypt registered [ 568.995774] Key type .llcrypt registered [ 569.142916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 569.156641] alg: No test for adler32 (adler32-zlib) [ 570.124496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.221609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 570.248219] LNet: 29574:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 570.256564] LNet: 29574:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 570.264185] LNet: 29574:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 570.287057] LNet: Message forwarding will be enabled in 2s [ 573.410271] LNet: Message forwarding enabled [ 573.890726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 573.921701] LNet: Message forwarding will stop in 2s to 6s [ 577.516389] LNet: Message forwarding disabled [ 578.429866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 578.447751] LNet: 29672:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 578.456828] LNet: 29672:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 578.462772] LNet: 29672:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 578.474904] LNet: Message forwarding will be enabled in 2s [ 578.601331] LNet: 29685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 579.187592] Key type .llcrypt unregistered [ 579.192206] Key type ._llcrypt unregistered [ 590.627311] Key type ._llcrypt registered [ 590.629743] Key type .llcrypt registered [ 591.980343] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 22:12:04 (1788833524) [ 592.453335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 592.466107] alg: No test for adler32 (adler32-zlib) [ 593.376386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.456697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 599.597368] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 22:12:12 (1788833532) [ 599.720586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.766942] LNet: 31078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 599.790463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.859733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 599.903432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 599.938611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 599.972703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 600.017342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 600.076032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 600.121895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 600.151518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 600.174895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 600.205489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 600.236915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 600.266835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 600.299604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 600.371076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 600.426339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 600.456245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 600.503696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 600.564624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 600.619941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 600.651705] LNetError: 31127:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 606.812689] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 22:12:20 (1788833540) [ 606.965961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.001890] LNet: 31696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 607.049243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.103789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 613.027818] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 22:12:26 (1788833546) [ 614.711355] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg412-server [ 616.511397] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 22:12:29 (1788833549) [ 616.676994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.735191] LNet: 32628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 616.749958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.792170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 616.815795] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 617.804907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[13-17]@tcp [ 617.850678] LNet: 32761:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 617.916877] LNetError: 32632:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.15@tcp is being used as a gateway but routing feature is not turned on [ 617.934334] LNetError: 32632:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.15@tcp has gone from up to down [ 624.428510] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 22:12:37 (1788833557) [ 626.226633] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg412-server [ 627.986065] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 22:12:41 (1788833561) [ 628.085161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 628.117165] LNet: 33689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.128913] LNetError: 33689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 628.179460] LNet: Removed LNI 192.168.204.12@tcp [ 628.239516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.340851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 628.375466] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 629.116056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[13-18/2]@tcp [ 629.144306] LNet: 33814:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 629.150843] LNet: 33814:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 629.167657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[13-18/2]@tcp [ 629.182367] LNetError: 33701:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.15@tcp is being used as a gateway but routing feature is not turned on [ 629.220700] LNetError: 33701:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 629.235871] LNetError: 33701:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.15@tcp has gone from up to down [ 629.246754] LNetError: 33701:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 634.857793] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 22:12:47 (1788833567) [ 634.947683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.967329] LNet: 34399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.974478] LNetError: 34399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 635.002533] LNet: Removed LNI 192.168.204.12@tcp [ 635.024127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.133849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 635.179135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.213974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.276871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 635.334039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.370824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.414733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 635.460558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.489299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.540757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 635.605989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.640614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.692328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 635.770446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.818739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.878706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 635.945456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.994549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.060624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 636.134524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.164631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.216647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 636.280962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.309410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.352640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 636.415551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.459510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.525467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 636.628266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.657985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.733485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 636.813316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.870717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.932533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 643.272594] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 22:12:56 (1788833576) [ 645.466245] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg412-server [ 647.039721] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 22:13:00 (1788833580) [ 648.939378] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg412-server [ 650.524217] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 22:13:03 (1788833583) [ 650.673022] LNet: 35808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.678592] LNet: 35808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 651.217616] Key type .llcrypt unregistered [ 651.220472] Key type ._llcrypt unregistered [ 657.026492] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 657.259178] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 657.275345] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 657.288767] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 660.009968] Key type ._llcrypt registered [ 660.014713] Key type .llcrypt registered [ 660.336789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 660.358335] alg: No test for adler32 (adler32-zlib) [ 661.328078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.417136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 661.470535] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 662.174309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 662.202949] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 664.226504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 664.258967] LNet: Removed LNI 10.1.2.3@tcp [ 664.420875] LNet: 36525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.435654] LNetError: 36525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.452686] LNet: Removed LNI 192.168.204.12@tcp [ 665.009683] Key type .llcrypt unregistered [ 665.013650] Key type ._llcrypt unregistered [ 675.482507] Key type ._llcrypt registered [ 675.484567] Key type .llcrypt registered [ 676.682721] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 22:13:29 (1788833609) [ 677.124389] Key type .llcrypt unregistered [ 677.126878] Key type ._llcrypt unregistered [ 682.877891] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 683.132234] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 683.134051] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 683.143843] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 685.613306] Key type ._llcrypt registered [ 685.615320] Key type .llcrypt registered [ 685.795679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 685.809675] alg: No test for adler32 (adler32-zlib) [ 686.748327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.814096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 686.836053] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 687.518569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 687.536626] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 689.981685] LNet: 38014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.987316] LNetError: 38014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 689.990910] LNet: Removed LNI 10.1.2.3@tcp [ 689.999734] LNet: Removed LNI 192.168.204.12@tcp [ 690.596912] Key type .llcrypt unregistered [ 690.600433] Key type ._llcrypt unregistered [ 700.216451] Key type ._llcrypt registered [ 700.224366] Key type .llcrypt registered [ 701.425124] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 22:13:54 (1788833634) [ 702.080845] Key type .llcrypt unregistered [ 702.089802] Key type ._llcrypt unregistered [ 708.005709] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 708.147738] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 708.158174] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 708.165290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 710.696364] Key type ._llcrypt registered [ 710.698403] Key type .llcrypt registered [ 710.921626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 710.939674] alg: No test for adler32 (adler32-zlib) [ 711.845602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.894738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 711.925817] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 713.863967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 713.889336] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 715.916818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 715.935587] LNet: Removed LNI 10.1.2.3@tcp [ 715.955105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 715.991071] LNet: Removed LNI 10.1.2.31@tcp [ 716.008133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 716.035869] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 716.855211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 716.880847] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 717.715665] LNet: 39922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.729051] LNetError: 39922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 717.742167] LNet: Removed LNI 10.1.2.31@tcp [ 717.755621] LNet: Removed LNI 10.1.2.3@tcp [ 718.416620] Key type .llcrypt unregistered [ 718.418487] Key type ._llcrypt unregistered [ 729.592936] Key type ._llcrypt registered [ 729.594506] Key type .llcrypt registered [ 730.793708] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 22:14:23 (1788833663) [ 731.305962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 731.331683] alg: No test for adler32 (adler32-zlib) [ 732.207400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.265145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 732.313470] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 733.213787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 733.244262] LNet: 40873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.251098] LNetError: 40873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 733.270275] LNet: Removed LNI 192.168.204.12@tcp [ 733.293593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.334600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.12@tcp [ 733.358173] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 734.023722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.12@tcp [ 734.063567] LNet: Removed LNI 192.168.204.12@tcp [ 734.160976] LNet: 41008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.958149] Key type .llcrypt unregistered [ 734.960280] Key type ._llcrypt unregistered [ 746.501655] Key type ._llcrypt registered [ 746.504220] Key type .llcrypt registered [ 747.677817] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 22:14:40 (1788833680) [ 749.740620] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg412-server [ 751.417275] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 22:14:44 (1788833684) [ 751.961386] Key type .llcrypt unregistered [ 751.964206] Key type ._llcrypt unregistered [ 758.095217] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 758.331989] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 758.338212] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 758.343072] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 760.820699] Key type ._llcrypt registered [ 760.822896] Key type .llcrypt registered [ 760.969382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 760.988461] alg: No test for adler32 (adler32-zlib) [ 761.920662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.029528] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 762.925342] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 764.906260] LNet: 42839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.923340] LNetError: 42839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 764.934514] LNet: Removed LNI 192.168.204.12@tcp1 [ 764.951331] LNet: Removed LNI 10.1.2.3@tcp2 [ 765.653395] Key type .llcrypt unregistered [ 765.662179] Key type ._llcrypt unregistered [ 777.025217] Key type ._llcrypt registered [ 777.027855] Key type .llcrypt registered [ 778.245917] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 22:15:11 (1788833711) [ 778.652212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 778.663245] alg: No test for adler32 (adler32-zlib) [ 779.577403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.619762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 779.636603] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 780.305603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.321663] LNet: 43807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.325714] LNetError: 43807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.341219] LNet: Removed LNI 192.168.204.12@tcp [ 780.372362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.446045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 780.467853] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 786.816299] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 22:15:19 (1788833719) [ 786.983363] LNet: 44392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.988720] LNetError: 44392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.010225] LNet: Removed LNI 192.168.204.12@tcp [ 787.576708] Key type .llcrypt unregistered [ 787.579994] Key type ._llcrypt unregistered [ 793.319594] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 793.569107] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 793.574997] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 793.579876] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 796.087589] Key type ._llcrypt registered [ 796.090210] Key type .llcrypt registered [ 796.286614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 796.299188] alg: No test for adler32 (adler32-zlib) [ 797.231297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.279388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.308038] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 797.989493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 798.009197] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.425772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.452896] LNet: 45136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.463617] LNetError: 45136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 800.471888] LNet: Removed LNI 10.1.2.3@tcp [ 800.483168] LNet: Removed LNI 192.168.204.12@tcp [ 800.503225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.605357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 800.632207] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 800.636124] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 801.317085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.360460] LNet: 45163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.368843] LNetError: 45163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 801.376878] LNetError: 45163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 801.386472] LNet: Removed LNI 10.1.2.3@tcp [ 801.404674] LNet: Removed LNI 192.168.204.12@tcp [ 801.432847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.534797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 801.585692] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 801.596316] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 802.033779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 802.052533] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 803.072389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.100366] LNet: 45326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.104971] LNetError: 45326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 803.109275] LNetError: 45326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 803.114560] LNet: Removed LNI 10.1.2.3@tcp [ 803.116244] LNet: Removed LNI 192.168.204.12@tcp [ 803.125687] LNet: Removed LNI 192.168.204.12@tcp2 [ 803.140394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.225954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 803.272822] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 803.284232] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 803.293066] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 810.593358] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 22:15:43 (1788833743) [ 810.774423] LNet: 45923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.778341] LNetError: 45923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 810.784863] LNetError: 45923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 810.799211] LNet: Removed LNI 10.1.2.3@tcp [ 810.804252] LNet: Removed LNI 192.168.204.12@tcp [ 810.811280] LNet: Removed LNI 192.168.204.12@tcp2 [ 811.450956] Key type .llcrypt unregistered [ 811.452965] Key type ._llcrypt unregistered [ 817.471535] Key type ._llcrypt registered [ 817.473458] Key type .llcrypt registered [ 817.678657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 817.714332] alg: No test for adler32 (adler32-zlib) [ 818.629890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.674409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 818.695949] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 819.357928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.377510] LNet: 46315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.382912] LNetError: 46315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.392796] LNet: Removed LNI 192.168.204.12@tcp [ 819.418496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.467421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 819.489668] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 819.952723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.977880] LNet: 46345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.982095] LNetError: 46345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.992361] LNet: Removed LNI 192.168.204.12@tcp [ 820.020839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.109970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 820.141716] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 820.614244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.638692] LNet: Removed LNI 192.168.204.12@tcp [ 820.658051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.722733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 820.744497] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 821.265731] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 821.406055] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 821.412896] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 821.419806] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 823.819113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 823.848377] LNet: 46455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.865266] LNet: 46455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 823.873820] LNetError: 46455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.884419] LNetError: 46455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 823.900682] LNet: Removed LNI 192.168.204.12@tcp [ 823.927752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.002951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 824.064413] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 830.692359] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 22:16:03 (1788833763) [ 831.025374] LNet: 47052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.032936] LNetError: 47052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 831.071387] LNet: Removed LNI 192.168.204.12@tcp [ 831.645866] Key type .llcrypt unregistered [ 831.649101] Key type ._llcrypt unregistered [ 837.974455] Key type ._llcrypt registered [ 837.979147] Key type .llcrypt registered [ 838.351835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 838.382934] alg: No test for adler32 (adler32-zlib) [ 839.315896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.377819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.411556] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 840.263344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.284898] LNet: 47448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.293842] LNetError: 47448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.310087] LNet: Removed LNI 192.168.204.12@tcp [ 840.346927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.409511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 840.441814] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 846.952655] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 22:16:20 (1788833780) [ 847.347261] LNet: 48040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.353606] LNetError: 48040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.374456] LNet: Removed LNI 192.168.204.12@tcp [ 847.975708] Key type .llcrypt unregistered [ 847.978755] Key type ._llcrypt unregistered [ 855.004920] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 855.177478] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 855.186632] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 855.189370] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 857.791781] Key type ._llcrypt registered [ 857.797600] Key type .llcrypt registered [ 858.023417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 858.039096] alg: No test for adler32 (adler32-zlib) [ 858.968979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.024542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 859.070961] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 859.956499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.974473] LNet: 48484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.991432] LNetError: 48484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.005987] LNet: Removed LNI 192.168.204.12@tcp [ 860.038878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.135552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 860.167406] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 866.787974] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 22:16:39 (1788833799) [ 866.880521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.935582] LNet: 49076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.944726] LNetError: 49076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.970412] LNet: Removed LNI 192.168.204.12@tcp [ 867.001148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.101168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 873.317474] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 22:16:46 (1788833806) [ 873.415490] LNet: 49655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.219604] Key type .llcrypt unregistered [ 874.221315] Key type ._llcrypt unregistered [ 880.639742] Key type ._llcrypt registered [ 880.641966] Key type .llcrypt registered [ 880.821895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 880.839863] alg: No test for adler32 (adler32-zlib) [ 881.761385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.854399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 881.906576] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 882.906097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.922217] LNet: 50059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.926984] LNetError: 50059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.949342] LNet: Removed LNI 192.168.204.12@tcp [ 882.998966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.079997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 883.117849] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 889.534969] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 22:17:02 (1788833822) [ 889.667427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.691211] LNet: 50647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.700879] LNetError: 50647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.725612] LNet: Removed LNI 192.168.204.12@tcp [ 889.759778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.846770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 895.265452] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 22:17:08 (1788833828) [ 896.758309] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 898.175742] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 22:17:11 (1788833831) [ 899.333463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 900.792351] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 22:17:14 (1788833834) [ 901.887373] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 903.174136] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 22:17:16 (1788833836) [ 904.275607] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 905.509448] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 22:17:18 (1788833838) [ 906.918461] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 908.264303] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 22:17:21 (1788833841) [ 908.406804] LNet: 52701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.081871] Key type .llcrypt unregistered [ 909.086497] Key type ._llcrypt unregistered [ 914.420460] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 914.553190] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 914.556205] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 914.559394] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 917.024499] Key type ._llcrypt registered [ 917.026330] Key type .llcrypt registered [ 917.163088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 917.181828] alg: No test for adler32 (adler32-zlib) [ 918.053388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.171052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 918.202635] LNet: Added LNI 192.168.204.12@tcp [8/512/0/180] [ 918.213382] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 918.221717] LNet: 53023:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 918.346367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.382621] LNet: 53042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.389211] LNetError: 53042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 918.401478] LNet: Removed LNI 10.1.2.3@tcp [ 918.408632] LNetError: 53042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.424122] LNet: Removed LNI 192.168.204.12@tcp [ 918.472116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.575083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 918.603263] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 918.666946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.690974] LNet: 53066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.698438] LNetError: 53066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.710472] LNet: Removed LNI 192.168.204.12@tcp [ 918.739922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.875639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 918.914130] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 918.932401] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 918.942332] LNet: 53074:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 919.078786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.114263] LNet: 53092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.127255] LNetError: 53092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 919.142424] LNet: Removed LNI 10.1.2.3@tcp [ 919.144533] LNetError: 53092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.159397] LNet: Removed LNI 192.168.204.12@tcp [ 919.199176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.255769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 919.313523] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 919.328168] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.333858] LNet: 53100:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 919.502806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.522839] LNet: 53126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.528089] LNetError: 53126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 919.531851] LNet: Removed LNI 10.1.2.3@tcp [ 919.533879] LNetError: 53126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.547934] LNet: Removed LNI 192.168.204.12@tcp [ 919.565099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.619105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 919.640055] LNet: Added LNI 192.168.204.12@tcp [8/256/0/182] [ 919.644241] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 919.702621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.716777] LNet: 53154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.721927] LNetError: 53154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 919.730533] LNet: Removed LNI 10.1.2.3@tcp [ 919.732868] LNetError: 53154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.752811] LNet: Removed LNI 192.168.204.12@tcp [ 919.785524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.843302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 919.864836] LNet: Added LNI 192.168.204.12@tcp [8/256/0/182] [ 919.869914] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 919.874940] LNet: 53162:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 919.936780] LNet: Removed LNI 10.1.2.3@tcp [ 919.951864] LNet: Removed LNI 192.168.204.12@tcp [ 920.603837] Key type .llcrypt unregistered [ 920.610451] Key type ._llcrypt unregistered [ 930.696930] Key type ._llcrypt registered [ 930.699184] Key type .llcrypt registered [ 931.864927] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 22:17:44 (1788833864) [ 933.279229] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 934.915658] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 22:17:47 (1788833867) [ 935.250030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 935.271603] alg: No test for adler32 (adler32-zlib) [ 936.266585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.341740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 936.365462] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 937.244600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.12 [ 937.277578] LNet: 54423:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 937.295050] LNet: Removed LNI 192.168.204.12@tcp [ 943.094888] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 22:17:56 (1788833876) [ 943.190527] LNet: 54984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.203978] LNetError: 54984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.216786] LNet: Removed LNI 192.168.204.12@tcp [ 943.774440] Key type .llcrypt unregistered [ 943.778069] Key type ._llcrypt unregistered [ 949.788495] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 949.997468] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 952.665433] Key type ._llcrypt registered [ 952.667940] Key type .llcrypt registered [ 952.822327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 952.839517] alg: No test for adler32 (adler32-zlib) [ 954.208573] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 960.235100] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 22:18:13 (1788833893) [ 960.351177] LNet: 55897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.360606] LNetError: 55897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 960.387753] LNet: Removed LNI 10.1.2.3@tcp [ 960.999704] Key type .llcrypt unregistered [ 961.006724] Key type ._llcrypt unregistered [ 967.458917] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 967.599392] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 967.607677] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 967.610267] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 970.281399] Key type ._llcrypt registered [ 970.285360] Key type .llcrypt registered [ 970.476859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 970.504818] alg: No test for adler32 (adler32-zlib) [ 971.444777] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 977.074315] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 22:18:29 (1788833909) [ 978.584488] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 980.519211] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 22:18:33 (1788833913) [ 980.591878] LNet: 57079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.600583] LNetError: 57079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 980.624474] LNet: Removed LNI 10.1.2.3@tcp [ 981.256460] Key type .llcrypt unregistered [ 981.258078] Key type ._llcrypt unregistered [ 986.065960] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 986.310222] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 986.316074] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 986.329326] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 989.021793] Key type ._llcrypt registered [ 989.024390] Key type .llcrypt registered [ 989.146267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 989.165923] alg: No test for adler32 (adler32-zlib) [ 990.108860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.230466] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 999.955101] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 22:18:53 (1788833933) [ 1000.367736] LNet: 58312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.383781] LNetError: 58312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1000.428754] LNet: Removed LNI 10.1.2.3@tcp [ 1001.143387] Key type .llcrypt unregistered [ 1001.149349] Key type ._llcrypt unregistered [ 1007.082182] Key type ._llcrypt registered [ 1007.086770] Key type .llcrypt registered [ 1007.357800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1007.375092] alg: No test for adler32 (adler32-zlib) [ 1008.439267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1008.494424] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1016.164178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1016.826468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_interrupt -r 1 [ 1016.863685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_interrupt -r 1 [ 1016.902591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1016.921670] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1017.152659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1017.177907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1017.208377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1017.264645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1017.321325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1017.353808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1017.404193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1017.488028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1017.558490] LNet: 58746:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1017.586691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1017.632870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1017.674653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1017.695432] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1017.701619] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1017.733308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1017.778246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1017.831581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1017.881628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1018.506944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_dropped -r 1 [ 1018.554215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_dropped -r 1 [ 1018.595861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1018.783171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1018.815407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1018.860579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1018.892727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1018.929934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1018.971402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1019.012417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1019.045255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1019.076760] LNet: 58855:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1019.102541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1019.131746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1019.159853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1019.209734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1019.233305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1019.257302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.280772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1019.736457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_aborted -r 1 [ 1019.770943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_aborted -r 1 [ 1019.799858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1019.814144] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1019.819190] LNet: Skipped 1 previous similar message [ 1020.048824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1020.090443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1020.154861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1020.263421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1020.342201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1020.401415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1020.462519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1020.550425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1020.637584] LNet: 58965:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1020.712756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1020.785486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1020.862406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1020.907819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1020.954298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1021.003318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1021.077572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1021.689855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_no_route -r 1 [ 1021.744982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_no_route -r 1 [ 1021.794550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1021.826983] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1022.139274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1022.244123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1022.337436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1022.384542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1022.406527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1022.465722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1022.517231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1022.556499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1022.602338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1022.652710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1022.691401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1022.734370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1022.791201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1022.827993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1022.860966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1023.402786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_timeout -r 1 [ 1023.426254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_timeout -r 1 [ 1023.450052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1023.634677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1023.666293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1023.686370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1023.717690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1023.741560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.760373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1023.779489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.805463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.821788] LNet: 59183:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1023.825411] LNet: 59183:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1023.837200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1023.858707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1023.885425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1023.921685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1023.948488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1023.967401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1023.987736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1024.574302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_error -r 1 [ 1024.596266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_error -r 1 [ 1024.640675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1024.878422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1024.933809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1024.969725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1025.038870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1025.083104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1025.128444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1025.180480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1025.240990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1025.273481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1025.319181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1025.340605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1025.367353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1025.406436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1025.452456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1025.486939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1025.821104] LNet: 59333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.826799] LNetError: 59333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1025.855506] LNet: Removed LNI 192.168.204.12@tcp [ 1026.292839] Key type .llcrypt unregistered [ 1026.296516] Key type ._llcrypt unregistered [ 1044.521401] Key type ._llcrypt registered [ 1044.529853] Key type .llcrypt registered [ 1045.751026] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 22:19:38 (1788833978) [ 1046.545298] Key type .llcrypt unregistered [ 1046.547214] Key type ._llcrypt unregistered [ 1053.736503] Key type ._llcrypt registered [ 1053.738298] Key type .llcrypt registered [ 1053.956712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1053.986547] alg: No test for adler32 (adler32-zlib) [ 1054.958166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1054.997407] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1063.442144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1074.003302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1074.057637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1074.085290] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1075.555966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_interrupt -r 1 [ 1075.585517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_interrupt -r 1 [ 1075.630369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_interrupt -r 1 [ 1075.666599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_interrupt -r 1 [ 1075.718945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_interrupt -r 1 [ 1075.758456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_interrupt -r 1 [ 1075.804405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_interrupt -r 1 [ 1075.864933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_interrupt -r 1 [ 1075.899524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1075.921187] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1075.926756] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1076.166311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1076.207718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1076.234991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1076.259947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1076.289583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1076.315522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1076.339909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1076.365571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1076.380453] LNet: 60798:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1076.394717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1076.423755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1076.448276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1076.476779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1076.498975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1076.526348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1076.561486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1076.995675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_dropped -r 1 [ 1077.028391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_dropped -r 1 [ 1077.061869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_dropped -r 1 [ 1077.085140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_dropped -r 1 [ 1077.112895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_dropped -r 1 [ 1077.147852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_dropped -r 1 [ 1077.183079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_dropped -r 1 [ 1077.226341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_dropped -r 1 [ 1077.260097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1077.278798] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1077.287347] LNet: Skipped 2 previous similar messages [ 1077.584804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1077.643943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1077.677067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1077.720282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1077.761784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1077.789243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1077.848702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1077.894290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1077.937661] LNet: 60919:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1077.975285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1078.018374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1078.055355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1078.081772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1078.133370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1078.171235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1078.230253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1078.773152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_aborted -r 1 [ 1078.810415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_aborted -r 1 [ 1078.838611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_aborted -r 1 [ 1078.868059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_aborted -r 1 [ 1078.902812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_aborted -r 1 [ 1078.934559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_aborted -r 1 [ 1078.982912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_aborted -r 1 [ 1079.062557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_aborted -r 1 [ 1079.116229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1079.134179] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1079.139667] LNet: Skipped 2 previous similar messages [ 1079.413995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1079.437151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1079.487865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1079.523903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1079.566929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1079.611375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1079.638279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1079.673748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1079.693853] LNet: 61041:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1079.712782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1079.761690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1079.793633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1079.826729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1079.882145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1079.925860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1079.957611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1080.570376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_no_route -r 1 [ 1080.603457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_no_route -r 1 [ 1080.637404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_no_route -r 1 [ 1080.676075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_no_route -r 1 [ 1080.715820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_no_route -r 1 [ 1080.749098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_no_route -r 1 [ 1080.802204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_no_route -r 1 [ 1080.867401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_no_route -r 1 [ 1080.939290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1081.233415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1081.274685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1081.315772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1081.348979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1081.393213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1081.432202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1081.463203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1081.493822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1081.532154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1081.578602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1081.619562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1081.659736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1081.717248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1081.748718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1081.783853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1082.230931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_timeout -r 1 [ 1082.272958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_timeout -r 1 [ 1082.314389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_timeout -r 1 [ 1082.350126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_timeout -r 1 [ 1082.393095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_timeout -r 1 [ 1082.425879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_timeout -r 1 [ 1082.485229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_timeout -r 1 [ 1082.534417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_timeout -r 1 [ 1082.575789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1082.600112] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1082.606618] LNet: Skipped 5 previous similar messages [ 1082.822693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1082.855466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1082.881944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1082.913874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1082.943868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1082.981475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1083.015455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1083.058949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1083.088882] LNet: 61283:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1083.095927] LNet: 61283:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1083.124055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1083.179805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1083.234867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1083.292222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1083.343294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1083.416670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1083.481410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1084.017942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_error -r 1 [ 1084.057711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_error -r 1 [ 1084.091709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_error -r 1 [ 1084.154810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_error -r 1 [ 1084.223579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_error -r 1 [ 1084.268727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_error -r 1 [ 1084.308986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_error -r 1 [ 1084.347208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_error -r 1 [ 1084.385364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1084.696354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1084.734537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1084.774881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1084.813517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1084.832841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1084.865429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1084.896127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1084.923383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1084.947970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1084.967562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1084.997573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1085.040783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1085.078460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1085.110193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1085.151133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1088.976986] LNet: 61481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.988557] LNetError: 61481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1088.999420] LNet: Removed LNI 192.168.204.12@tcp [ 1089.011082] LNet: Removed LNI 192.168.204.12@tcp1 [ 1089.403448] Key type .llcrypt unregistered [ 1089.407996] Key type ._llcrypt unregistered [ 1107.304589] Key type ._llcrypt registered [ 1107.307606] Key type .llcrypt registered [ 1108.760779] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 22:20:41 (1788834041) [ 1109.467474] Key type .llcrypt unregistered [ 1109.471448] Key type ._llcrypt unregistered [ 1116.345548] Key type ._llcrypt registered [ 1116.348702] Key type .llcrypt registered [ 1116.594940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1116.610772] alg: No test for adler32 (adler32-zlib) [ 1117.531869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1117.561810] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1126.310776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1127.279607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e remote_dropped -r 1 [ 1127.326969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e remote_dropped -r 1 [ 1127.430322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1127.481858] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1127.745972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1127.777274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1127.816209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1127.860421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1127.920912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1127.959394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1128.020985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1128.061829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1128.086213] LNet: 62728:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1128.121914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1128.161390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1128.206619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1128.243740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1128.296211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1128.337691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1128.375420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1128.901170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e remote_error -r 1 [ 1128.935368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e remote_error -r 1 [ 1128.978452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1129.017130] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1129.256927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1129.293733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1129.322683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1129.349459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1129.375599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1129.399768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1129.422497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1129.482518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1129.502823] LNet: 62838:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1129.522275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1129.567128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1129.616914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1129.648912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1129.687235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.740721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.811184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1130.547917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e remote_timeout -r 1 [ 1130.589710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e remote_timeout -r 1 [ 1130.622643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1130.640723] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1130.826864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1130.866681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1130.905816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1130.942245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1130.981853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1131.018837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1131.057316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1131.112251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1131.140414] LNet: 62947:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1131.165377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1131.236204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1131.272498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1131.317673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1131.347352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1131.374638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1131.419344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1131.748515] LNet: 62988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1131.755919] LNetError: 62988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1131.796798] LNet: Removed LNI 192.168.204.12@tcp [ 1132.390702] Key type .llcrypt unregistered [ 1132.392406] Key type ._llcrypt unregistered [ 1150.012745] Key type ._llcrypt registered [ 1150.015520] Key type .llcrypt registered [ 1151.448392] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 22:21:24 (1788834084) [ 1152.344323] Key type .llcrypt unregistered [ 1152.347658] Key type ._llcrypt unregistered [ 1158.649037] Key type ._llcrypt registered [ 1158.655414] Key type .llcrypt registered [ 1158.924962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1158.950341] alg: No test for adler32 (adler32-zlib) [ 1159.879338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1159.912530] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1167.462421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1176.537394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1176.596506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1176.641983] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1178.276372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e remote_dropped -r 1 [ 1178.308444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e remote_dropped -r 1 [ 1178.338928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e remote_dropped -r 1 [ 1178.371523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e remote_dropped -r 1 [ 1178.415082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e remote_dropped -r 1 [ 1178.450180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e remote_dropped -r 1 [ 1178.492116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e remote_dropped -r 1 [ 1178.557924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e remote_dropped -r 1 [ 1178.625749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1178.647334] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1178.654452] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.112@tcp [ 1178.962500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.029457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1179.088567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.131299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.156679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.180199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1179.205234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1179.229948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1179.253879] LNet: 64450:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1179.274798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1179.334315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1179.385226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1179.415893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.455426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.497931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1179.545938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1179.582125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.223059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e remote_error -r 1 [ 1180.292173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e remote_error -r 1 [ 1180.363371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e remote_error -r 1 [ 1180.401383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e remote_error -r 1 [ 1180.426729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e remote_error -r 1 [ 1180.477781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e remote_error -r 1 [ 1180.524826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e remote_error -r 1 [ 1180.566895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e remote_error -r 1 [ 1180.610711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1180.640109] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1180.646730] LNet: Skipped 2 previous similar messages [ 1180.810741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.837962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1180.868892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1180.898806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1180.925872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1180.957588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.994746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.031211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.047563] LNet: 64574:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1181.064535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.108593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.140629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.178893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.215960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.250292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.304314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1181.366465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.953281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e remote_timeout -r 1 [ 1181.987240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e remote_timeout -r 1 [ 1182.022415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e remote_timeout -r 1 [ 1182.067643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e remote_timeout -r 1 [ 1182.104534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e remote_timeout -r 1 [ 1182.138188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e remote_timeout -r 1 [ 1182.175224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e remote_timeout -r 1 [ 1182.204378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e remote_timeout -r 1 [ 1182.237818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1182.261859] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1182.456231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.480315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.501795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.528393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.554740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.586534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.621167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.653704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.677189] LNet: 64697:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.701380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.737728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.780111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.817661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.853352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.887990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.941691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1182.994900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1188.102183] LNet: 64775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1188.119580] LNetError: 64775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1188.126388] LNet: Removed LNI 192.168.204.12@tcp [ 1188.167634] LNet: Removed LNI 192.168.204.12@tcp1 [ 1188.538780] Key type .llcrypt unregistered [ 1188.544504] Key type ._llcrypt unregistered [ 1208.140971] Key type ._llcrypt registered [ 1208.143451] Key type .llcrypt registered [ 1209.289196] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 22:22:22 (1788834142) [ 1210.941493] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1213.535320] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 22:22:25 (1788834145) [ 1214.536410] Key type .llcrypt unregistered [ 1214.539549] Key type ._llcrypt unregistered [ 1221.701102] Key type ._llcrypt registered [ 1221.705409] Key type .llcrypt registered [ 1222.100683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1222.131654] alg: No test for adler32 (adler32-zlib) [ 1223.155396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1223.217328] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1230.682413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1231.339490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e network_timeout -r 1 [ 1231.388762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e network_timeout -r 1 [ 1231.440449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1231.468906] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1231.697654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1231.746244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1231.777117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1231.819764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1231.862046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1231.889857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1231.922730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1231.962541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1231.995433] LNet: 66315:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1232.021510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1232.081637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1232.140898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1232.205567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1232.248396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1232.278675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1232.633776] LNet: 66354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1232.647476] LNetError: 66354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1232.686468] LNet: Removed LNI 192.168.204.12@tcp [ 1233.213531] Key type .llcrypt unregistered [ 1233.218975] Key type ._llcrypt unregistered [ 1251.646516] Key type ._llcrypt registered [ 1251.648814] Key type .llcrypt registered [ 1251.880220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1251.899541] alg: No test for adler32 (adler32-zlib) [ 1252.829723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1252.856871] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1260.180277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1269.036051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1269.105940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1269.132823] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1270.371239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e network_timeout -r 1 [ 1270.427770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e network_timeout -r 1 [ 1270.475098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e network_timeout -r 1 [ 1270.520606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e network_timeout -r 1 [ 1270.576729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e network_timeout -r 1 [ 1270.613335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e network_timeout -r 1 [ 1270.644188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e network_timeout -r 1 [ 1270.683334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e network_timeout -r 1 [ 1270.728623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1270.762380] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1271.014893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1271.041619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1271.069923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1271.119075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1271.152581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1271.190275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1271.222863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1271.265327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1271.298669] LNet: 67228:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1271.316539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1271.327985] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1271.334955] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.112@tcp [ 1271.336188] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1271.361474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1271.409952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1271.449055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1271.517816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1271.575804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1275.854095] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1275.861196] LNet: Skipped 1 previous similar message [ 1276.149125] LNet: 67302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1276.156818] LNetError: 67302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.215759] LNet: Removed LNI 192.168.204.12@tcp [ 1277.217501] LNetError: 67302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.234461] LNet: Removed LNI 192.168.204.12@tcp1 [ 1277.572904] Key type .llcrypt unregistered [ 1277.575586] Key type ._llcrypt unregistered [ 1293.355596] Key type ._llcrypt registered [ 1293.362185] Key type .llcrypt registered [ 1294.640128] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 22:23:47 (1788834227) [ 1295.007661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1295.027370] alg: No test for adler32 (adler32-zlib) [ 1295.975677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1296.034828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1296.068821] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1296.723191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1296.749092] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1297.644805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 1297.704886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1297.749866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1297.789966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1297.821783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1297.852825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1297.888426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.12@tcp [ 1297.914610] LNetError: 68400:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.12@tcp to recovery [ 1298.079479] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1298.083839] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1300.146540] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1300.345178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1304.223626] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1304.944374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1304.978433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1305.022733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1305.034443] LNet: 68507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1305.038421] LNetError: 68507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1305.043453] LNet: Removed LNI 192.168.204.12@tcp [ 1305.057842] LNet: Removed LNI 192.168.204.12@tcp1 [ 1305.081308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1305.135141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1305.159111] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1305.804323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1305.817608] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1306.455727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 1306.489410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.540368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1306.580288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1306.607266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1306.643280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1306.678381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1306.713208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.12@tcp [ 1306.737140] LNetError: 68779:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.12@tcp to recovery [ 1307.169636] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1310.257510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1313.311480] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1313.317264] LNet: Skipped 1 previous similar message [ 1318.182968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1318.218366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1318.252719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1324.040338] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 22:24:17 (1788834257) [ 1324.133513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1324.154036] LNet: 69490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.164800] LNetError: 69490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.174749] LNetError: 69490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1324.180184] LNet: Removed LNI 192.168.204.12@tcp [ 1324.194645] LNet: Removed LNI 192.168.204.12@tcp1 [ 1324.226919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.281410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1324.305472] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1324.867271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1324.885653] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1325.611242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 1325.664767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1325.700829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1325.744054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1325.787888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1325.812509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1325.837483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.12@tcp --health 0 [ 1325.935603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1326.303310] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.12@tcp [ 1326.304313] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1326.316115] LNet: Skipped 1 previous similar message [ 1341.216804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1341.344607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1341.380745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.422866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.12@tcp --health 500 [ 1341.561206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1348.301626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1348.367378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1348.386153] LNet: 69988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1348.392158] LNetError: 69988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1348.396926] LNetError: 69988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1348.402954] LNet: Removed LNI 192.168.204.12@tcp [ 1348.416885] LNet: Removed LNI 192.168.204.12@tcp1 [ 1348.457710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1348.537032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1348.579994] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1349.126754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1349.153768] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1349.970744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 1349.999229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1350.051458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1350.087844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1350.123765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1350.166246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1350.192923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1350.227267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.12@tcp --health 0 [ 1351.584304] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1351.593675] LNet: Skipped 3 previous similar messages [ 1352.651372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1362.702408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1362.725035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1362.752269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1369.009175] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 22:25:02 (1788834302) [ 1369.458910] LNet: 71045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1369.463814] LNetError: 71045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1369.467798] LNetError: 71045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1369.473209] LNet: Removed LNI 192.168.204.12@tcp [ 1369.482833] LNet: Removed LNI 192.168.204.12@tcp1 [ 1370.178849] Key type .llcrypt unregistered [ 1370.181571] Key type ._llcrypt unregistered [ 1375.644929] Key type ._llcrypt registered [ 1375.647501] Key type .llcrypt registered [ 1375.885718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1375.908626] alg: No test for adler32 (adler32-zlib) [ 1376.786790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1376.808848] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1384.684334] Lustre: DEBUG MARKER: Initial discovery [ 1384.725588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.112@tcp [ 1386.648104] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1386.695668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1386.754313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.112@tcp [ 1386.798566] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1386.833496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1386.891622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.112@tcp -r 1 -m GET -l 3 [ 1386.935262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.112@tcp -r 1 -m GET -e local_error [ 1386.976966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.112@tcp -r 1 -m PUT -l 6 [ 1387.040269] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1388.398899] Lustre: DEBUG MARKER: Force oleg412-client.virtnet to discover 192.168.204.112@tcp (in background) [ 1388.481783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.112@tcp [ 1389.886734] Lustre: DEBUG MARKER: Force oleg412-server to discover 192.168.204.12@tcp [ 1391.593434] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1391.959332] Lustre: DEBUG MARKER: Wait for 71798 [ 1393.482784] Lustre: DEBUG MARKER: Finished wait on 71798 [ 1393.621969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1393.655500] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1393.672774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1393.951430] LNet: 72255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.962811] LNetError: 72255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.987130] LNet: Removed LNI 192.168.204.12@tcp [ 1394.357695] Key type .llcrypt unregistered [ 1394.361070] Key type ._llcrypt unregistered [ 1411.085837] Key type ._llcrypt registered [ 1411.091842] Key type .llcrypt registered [ 1412.379487] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 22:25:45 (1788834345) [ 1413.041176] Key type .llcrypt unregistered [ 1413.047237] Key type ._llcrypt unregistered [ 1419.539047] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1419.657832] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1419.665370] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1419.671048] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1422.272240] Key type ._llcrypt registered [ 1422.276151] Key type .llcrypt registered [ 1422.509378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1422.538464] alg: No test for adler32 (adler32-zlib) [ 1423.452427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1423.493588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1423.513322] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1424.247907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1424.287272] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1431.345966] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 22:26:04 (1788834364) [ 1431.434481] LNet: 74314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.448770] LNetError: 74314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1431.455767] LNet: Removed LNI 10.1.2.3@tcp [ 1431.485622] LNet: Removed LNI 192.168.204.12@tcp [ 1432.061593] Key type .llcrypt unregistered [ 1432.066768] Key type ._llcrypt unregistered [ 1439.638331] Key type ._llcrypt registered [ 1439.641742] Key type .llcrypt registered [ 1439.882446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1439.901465] alg: No test for adler32 (adler32-zlib) [ 1440.811191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1440.891389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1440.927471] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1441.923337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1441.960794] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1443.480212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.12@tcp [ 1444.575687] LNetError: Unexpected error -22 connecting to 192.168.204.12@tcp at host 192.168.204.12:988 [ 1450.900772] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 22:26:23 (1788834383) [ 1451.022028] LNet: 75446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.035792] LNetError: 75446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1451.043528] LNet: Removed LNI 10.1.2.3@tcp [ 1451.064567] LNet: Removed LNI 192.168.204.12@tcp [ 1451.756830] Key type .llcrypt unregistered [ 1451.761316] Key type ._llcrypt unregistered [ 1457.222668] Key type ._llcrypt registered [ 1457.224914] Key type .llcrypt registered [ 1457.390588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1457.409649] alg: No test for adler32 (adler32-zlib) [ 1458.382253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1458.437193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1458.482783] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1459.187127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1459.213617] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 1459.916118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.12@tcp1 --nid 192.168.204.12@tcp2 [ 1464.447952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp1 192.168.204.12@tcp2 [ 1464.466689] LNetError: 76286:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.12@tcp1 to 192.168.204.12@tcp2 - Route table may be misconfigured [ 1464.500832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp2 192.168.204.12@tcp1 [ 1470.221656] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 22:26:43 (1788834403) [ 1470.317646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1470.342673] LNet: 76853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1470.349106] LNetError: 76853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1470.354968] LNet: Removed LNI 192.168.204.12@tcp1 [ 1471.398911] LNet: Removed LNI 192.168.204.12@tcp2 [ 1471.441375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.500248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1471.542416] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1472.231348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1472.248786] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1472.940627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 1472.978985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp 192.168.204.12@tcp [ 1473.010914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp1 192.168.204.12@tcp1 [ 1473.042533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.12@tcp -d 192.168.204.12@tcp -e network_timeout [ 1473.079926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -e network_timeout [ 1473.133604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -e network_timeout [ 1473.171983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -e network_timeout [ 1473.216890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1473.245568] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1473.262629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1473.359803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1473.452441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1473.504688] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1479.137562] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 22:26:52 (1788834412) [ 1479.428911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1479.472435] LNet: 77722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.479170] LNetError: 77722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1479.487825] LNetError: 77722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1479.501670] LNet: Removed LNI 192.168.204.12@tcp [ 1479.554672] LNet: Removed LNI 192.168.204.12@tcp1 [ 1479.630080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.753336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1480.358804] Key type .llcrypt unregistered [ 1480.366500] Key type ._llcrypt unregistered [ 1490.431953] Key type ._llcrypt registered [ 1490.434200] Key type .llcrypt registered [ 1491.569256] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 22:27:04 (1788834424) [ 1491.883927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1491.893437] alg: No test for adler32 (adler32-zlib) [ 1492.796505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1493.946896] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1495.400077] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 22:27:08 (1788834428) [ 1495.480351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1495.527308] LNet: 78848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1495.558454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1495.630106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1495.653254] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1496.197624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1496.217631] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1497.001340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1497.054548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp1 [ 1497.084765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp1 [ 1502.212888] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 22:27:15 (1788834435) [ 1503.726503] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg412-server [ 1505.097775] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 22:27:18 (1788834438) [ 1506.511949] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg412-server [ 1508.336898] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 22:27:21 (1788834441) [ 1509.935379] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg412-server [ 1511.656838] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 22:27:24 (1788834444) [ 1513.467393] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg412-server [ 1515.114371] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 22:27:28 (1788834448) [ 1516.834329] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg412-server [ 1518.528955] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 22:27:31 (1788834451) [ 1520.039550] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg412-server [ 1521.500622] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 22:27:34 (1788834454) [ 1523.229393] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg412-server [ 1524.739528] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 22:27:38 (1788834458) [ 1526.501431] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg412-server [ 1528.218839] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 22:27:41 (1788834461) [ 1529.916718] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg412-server [ 1531.637568] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 22:27:44 (1788834464) [ 1531.719901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1531.739952] LNet: 82726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1531.754161] LNetError: 82726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1531.759767] LNet: Removed LNI 192.168.204.12@tcp [ 1532.774182] LNet: Removed LNI 192.168.204.12@tcp1 [ 1532.850874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1532.928588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1532.966662] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1533.650291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1533.751304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1533.963347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1533.976762] LNet: 82878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1533.988073] LNetError: 82878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.994502] LNetError: 82878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1534.009080] LNet: Removed LNI 192.168.204.12@tcp [ 1534.025818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.113635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1534.141425] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1534.807723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1535.010188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.030844] LNetError: 83023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.048182] LNet: Removed LNI 192.168.204.12@tcp [ 1535.064199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.143137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1535.175912] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1535.874483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1535.944150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1536.186134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1536.212518] LNet: 83170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.218545] LNet: 83170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1537.254462] LNet: Removed LNI 192.168.204.12@tcp [ 1537.289436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.348727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1537.366967] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1538.042293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1538.201985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.221668] LNetError: 83315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1538.226582] LNetError: 83315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1539.237770] LNet: Removed LNI 192.168.204.12@tcp [ 1539.264466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.349539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1539.379734] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1540.296435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1540.397784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1540.573124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.593713] LNet: 83463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.603960] LNet: 83463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1540.629159] LNet: Removed LNI 192.168.204.12@tcp [ 1540.662727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.724898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1540.747812] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1541.447330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1541.651889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.759799] LNet: Removed LNI 192.168.204.12@tcp [ 1542.793289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.885715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1542.909045] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1543.642247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1543.701964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1543.918428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.942977] LNetError: 83755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.954286] LNetError: 83755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1543.979345] LNet: Removed LNI 192.168.204.12@tcp [ 1543.995535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1544.029296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1544.050695] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1545.049458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1545.404810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.473682] LNet: Removed LNI 192.168.204.12@tcp [ 1546.526953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.607673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1546.659229] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1547.858271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1547.997367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1548.290186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1549.377908] LNet: Removed LNI 192.168.204.12@tcp [ 1549.429712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.528395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1549.566254] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1550.738422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1552.274831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.287439] LNet: 84203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.292304] LNet: 84203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1552.298350] LNetError: 84203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.308070] LNetError: 84203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1552.319976] LNet: Removed LNI 192.168.204.12@tcp [ 1552.336647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.374715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1552.388709] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1552.907772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1553.002281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1553.270774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1554.341892] LNet: Removed LNI 192.168.204.12@tcp [ 1554.401274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.494567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1554.532370] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1555.475316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1555.703894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1556.774688] LNet: Removed LNI 192.168.204.12@tcp [ 1556.874356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.008993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1557.058654] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1557.818131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1557.870248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.894182] LNet: Removed LNI 192.168.204.12@tcp [ 1557.947337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.059065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1558.100502] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1559.519628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1559.661481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.722413] LNet: Removed LNI 192.168.204.12@tcp [ 1559.758387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.844429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1559.956603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.028679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.156514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1560.289518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.326938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.378936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1560.484381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.519818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.573977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1560.594945] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1561.557625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.606345] LNet: Removed LNI 192.168.204.12@tcp [ 1561.631576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.702958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1569.136858] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 22:28:22 (1788834502) [ 1569.213658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.247690] LNet: 85509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.264939] LNet: 85509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1569.301683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.377221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1569.439315] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/180] [ 1570.620104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.634577] LNetError: 85656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.642028] LNetError: 85656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1570.662189] LNet: Removed LNI 192.168.204.12@tcp231 [ 1570.695508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.768556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1570.792352] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/180] [ 1570.799584] LNet: 85664:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1571.150033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1571.177612] LNet: Removed LNI 192.168.204.12@tcp231 [ 1571.192781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1571.223959] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/180] [ 1572.057844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.140909] LNet: Removed LNI 192.168.204.12@tcp231 [ 1572.177685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.221273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1572.258180] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/0] [ 1572.266258] LNet: 85816:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1572.718589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1572.742446] LNet: Removed LNI 192.168.204.12@tcp231 [ 1572.752290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1572.771974] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/0] [ 1573.691672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.725522] LNet: Removed LNI 192.168.204.12@tcp231 [ 1573.746745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.829479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1573.858627] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/60] [ 1573.867486] LNet: 85967:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1574.250727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1574.289910] LNet: Removed LNI 192.168.204.12@tcp231 [ 1574.321699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1574.384897] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/60] [ 1575.495763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.532546] LNet: Removed LNI 192.168.204.12@tcp231 [ 1575.550876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.606241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1575.629642] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/180] [ 1575.957359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1576.007749] LNet: Removed LNI 192.168.204.12@tcp231 [ 1576.029434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1576.085529] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/180] [ 1583.613787] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 22:28:36 (1788834516) [ 1583.734552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1583.807569] LNet: Removed LNI 192.168.204.12@tcp231 [ 1583.845506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.896112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1583.923086] LNet: Added LNI 192.168.204.12@tcp232 [8/256/0/180] [ 1584.638177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1584.694182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1584.731163] LNet: Removed LNI 192.168.204.12@tcp232 [ 1584.766513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1584.806377] LNet: Added LNI 192.168.204.12@tcp232 [8/256/0/180] [ 1585.594326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1585.682809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1585.720183] LNet: Removed LNI 192.168.204.12@tcp232 [ 1585.755184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1585.787735] LNet: Added LNI 192.168.204.12@tcp232 [8/256/0/180] [ 1586.887560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1586.973610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1587.013160] LNet: Removed LNI 192.168.204.12@tcp232 [ 1587.038694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1587.064696] LNet: Added LNI 192.168.204.12@tcp232 [8/256/0/180] [ 1587.864765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1587.906414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1587.938234] LNet: Removed LNI 192.168.204.12@tcp232 [ 1587.969575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1593.956435] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 22:28:46 (1788834526) [ 1594.888929] Key type .llcrypt unregistered [ 1594.892526] Key type ._llcrypt unregistered [ 1600.408334] Key type ._llcrypt registered [ 1600.410560] Key type .llcrypt registered [ 1600.613949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1600.633878] alg: No test for adler32 (adler32-zlib) [ 1601.577811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1601.617997] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1608.402167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1616.639533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.677625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1616.703437] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1618.331183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_interrupt -r 2 [ 1618.384981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_interrupt -r 2 [ 1618.431169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_interrupt -r 2 [ 1618.491132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_interrupt -r 2 [ 1618.532690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_interrupt -r 2 [ 1618.572502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_interrupt -r 2 [ 1618.607761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_interrupt -r 2 [ 1618.644711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_interrupt -r 2 [ 1618.684958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp 192.168.204.112@tcp [ 1618.711119] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1618.718722] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1619.005896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1619.051757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1619.107182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1619.146524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1619.179934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1619.214569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1619.256697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1619.310332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1619.343713] LNet: 88617:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1619.369653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1619.410748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1619.451879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1619.493977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1619.539678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1619.599852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1620.283813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_dropped -r 2 [ 1620.329921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_dropped -r 2 [ 1620.375726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_dropped -r 2 [ 1620.444595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_dropped -r 2 [ 1620.499774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_dropped -r 2 [ 1620.558955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_dropped -r 2 [ 1620.607638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_dropped -r 2 [ 1620.672217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_dropped -r 2 [ 1620.729884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp 192.168.204.112@tcp [ 1620.761458] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1621.097740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1621.134893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1621.179313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1621.218360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1621.250590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1621.277373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1621.304527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1621.343802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1621.368574] LNet: 88738:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1621.383271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1621.417527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1621.448523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1621.488903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1621.525403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1621.589349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1622.160567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_aborted -r 2 [ 1622.184939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_aborted -r 2 [ 1622.233551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_aborted -r 2 [ 1622.277377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_aborted -r 2 [ 1622.315890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_aborted -r 2 [ 1622.363708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_aborted -r 2 [ 1622.386849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_aborted -r 2 [ 1622.414759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_aborted -r 2 [ 1622.478646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp 192.168.204.112@tcp [ 1622.508672] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1622.766688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1622.794243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1622.817380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1622.842376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1622.866919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1622.902769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1622.928596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1622.958341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1622.972352] LNet: 88858:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1622.988889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1623.053773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1623.119525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1623.150710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1623.177408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1623.200715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1623.642329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_no_route -r 2 [ 1623.671699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_no_route -r 2 [ 1623.697678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_no_route -r 2 [ 1623.727196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_no_route -r 2 [ 1623.761463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_no_route -r 2 [ 1623.795591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_no_route -r 2 [ 1623.826937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_no_route -r 2 [ 1623.854463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_no_route -r 2 [ 1623.892352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp 192.168.204.112@tcp [ 1624.079529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1624.101581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1624.123360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1624.143816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1624.168771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1624.192948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1624.217898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1624.244788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1624.278721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1624.315690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1624.362959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1624.404567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1624.437066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1624.470771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1624.849966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_timeout -r 2 [ 1624.894960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_timeout -r 2 [ 1624.949359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_timeout -r 2 [ 1624.990632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_timeout -r 2 [ 1625.029828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_timeout -r 2 [ 1625.062610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_timeout -r 2 [ 1625.093860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_timeout -r 2 [ 1625.132423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_timeout -r 2 [ 1625.161260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp 192.168.204.112@tcp [ 1625.181189] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1625.185234] LNet: Skipped 1 previous similar message [ 1625.521500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1625.609893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1625.670941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1625.721441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1625.767618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1625.835150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1625.884244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1625.953572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1626.003764] LNet: 89098:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1626.020268] LNet: 89098:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1626.056631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1626.142936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1626.209519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1626.285961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1626.324803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1626.380636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1626.981119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e remote_dropped -r 2 [ 1627.033725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e remote_dropped -r 2 [ 1627.109032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e remote_dropped -r 2 [ 1627.161688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e remote_dropped -r 2 [ 1627.194704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e remote_dropped -r 2 [ 1627.223318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e remote_dropped -r 2 [ 1627.255563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e remote_dropped -r 2 [ 1627.294387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e remote_dropped -r 2 [ 1627.357675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp 192.168.204.112@tcp [ 1627.393679] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.112@tcp [ 1627.627624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1627.660292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1627.681736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1627.719769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1627.753630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1627.788374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1627.819075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1627.853683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1627.919544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1627.980817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1628.019639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1628.052855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1628.093645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1628.133854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1633.084427] LNet: 89293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.092105] LNetError: 89293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.098843] LNet: Removed LNI 192.168.204.12@tcp [ 1633.108888] LNet: Removed LNI 192.168.204.12@tcp1 [ 1633.565042] Key type .llcrypt unregistered [ 1633.572083] Key type ._llcrypt unregistered [ 1649.551932] Key type ._llcrypt registered [ 1649.556831] Key type .llcrypt registered [ 1651.089764] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 22:29:43 (1788834583) [ 1651.492500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1651.508914] alg: No test for adler32 (adler32-zlib) [ 1652.491328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.597267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1652.640057] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1653.586727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1653.605346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.12@tcp [ 1653.619122] LNetError: 90269:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.12@tcp to recovery [ 1653.679550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.12@tcp [ 1653.768334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1659.382888] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 22:29:52 (1788834592) [ 1659.543687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1659.576249] LNet: 90860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.586756] LNetError: 90860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1659.648922] LNet: Removed LNI 192.168.204.12@tcp [ 1659.672829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.717671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1659.751670] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1660.506036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1660.524542] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1661.614499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.12@tcp [ 1661.633652] LNetError: 91126:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.12@tcp to recovery [ 1661.727361] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1661.728671] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1662.882477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.12@tcp [ 1671.407729] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 22:30:04 (1788834604) [ 1673.130771] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg412-server [ 1675.003397] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 22:30:07 (1788834607) [ 1676.930256] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg412-server [ 1678.684844] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 22:30:11 (1788834611) [ 1680.608738] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg412-server [ 1682.907995] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 22:30:15 (1788834615) [ 1683.062723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.116391] LNet: 92744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.127627] LNetError: 92744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.135989] LNet: Removed LNI 192.168.204.12@tcp [ 1683.145042] LNet: Removed LNI 192.168.204.12@tcp1 [ 1683.232244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.300425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1683.342552] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1684.260349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1684.328528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1684.388956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1684.525793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1684.579147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1684.681791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1691.182105] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 22:30:24 (1788834624) [ 1691.248398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.270246] LNet: 93473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.280334] LNetError: 93473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.287705] LNetError: 93473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1691.313927] LNet: Removed LNI 192.168.204.12@tcp [ 1691.337965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.393833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1691.421862] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1698.154265] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 22:30:31 (1788834631) [ 1698.320928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.339537] LNet: 94184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1698.347730] LNetError: 94184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1698.364196] LNet: Removed LNI 192.168.204.12@tcp [ 1698.381176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.462629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1698.498526] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1704.631557] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 22:30:37 (1788834637) [ 1706.188109] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1708.073150] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 22:30:41 (1788834641) [ 1708.620878] LNet: 95218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.626379] LNetError: 95218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1708.645214] LNet: Removed LNI 192.168.204.12@tcp [ 1709.198799] Key type .llcrypt unregistered [ 1709.202541] Key type ._llcrypt unregistered [ 1715.841216] Key type ._llcrypt registered [ 1715.844661] Key type .llcrypt registered [ 1716.171659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1716.211307] alg: No test for adler32 (adler32-zlib) [ 1717.167758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1717.190463] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1723.541945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1728.704594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.112@tcp [ 1729.283854] LNet: 95633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.299131] LNetError: 95633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.364980] LNet: Removed LNI 192.168.204.12@tcp [ 1729.967839] Key type .llcrypt unregistered [ 1729.973873] Key type ._llcrypt unregistered [ 1744.381494] Key type ._llcrypt registered [ 1744.384894] Key type .llcrypt registered [ 1745.956421] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1747.689972] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1749.884460] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 22:31:22 (1788834682) [ 1751.259626] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1753.103237] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 22:31:26 (1788834686) [ 1755.335490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg412-server [ 1757.158644] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 22:31:30 (1788834690) [ 1759.173574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg412-server [ 1760.941106] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 22:31:34 (1788834694) [ 1762.487561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1762.516787] alg: No test for adler32 (adler32-zlib) [ 1763.661602] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 1763.906038] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1765.583447] Key type lgssc registered [ 1767.183760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1787.534450] Key type lgssc unregistered [ 1787.791220] LNet: 98804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.800743] LNetError: 98804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.824901] LNet: Removed LNI 192.168.204.12@tcp [ 1788.589361] Key type .llcrypt unregistered [ 1788.599293] Key type ._llcrypt unregistered [ 1804.254991] Key type ._llcrypt registered [ 1804.265221] Key type .llcrypt registered [ 1805.598806] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 22:32:18 (1788834738) [ 1806.022829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1806.034809] alg: No test for adler32 (adler32-zlib) [ 1807.057361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.176219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1807.220778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1807.301743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1807.381496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1813.011251] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 22:32:26 (1788834746) [ 1813.183963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.213770] LNet: 100213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.253295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.319263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1813.348960] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1814.026499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.044966] LNet: 100361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.053961] LNetError: 100361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.071537] LNet: Removed LNI 192.168.204.12@tcp [ 1814.094915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.150636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1814.188039] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1814.672455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.702344] LNetError: 100390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.730270] LNet: Removed LNI 192.168.204.12@tcp [ 1814.746624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.804751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1814.835255] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1821.223272] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 22:32:34 (1788834754) [ 1821.346079] LNet: 100972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.352584] LNet: 100972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1821.363752] LNetError: 100972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.385100] LNet: Removed LNI 192.168.204.12@tcp [ 1821.995594] Key type .llcrypt unregistered [ 1821.998630] Key type ._llcrypt unregistered [ 1827.873911] Key type ._llcrypt registered [ 1827.884273] Key type .llcrypt registered [ 1828.096303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1828.127373] alg: No test for adler32 (adler32-zlib) [ 1829.044016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.123322] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1829.127997] LNet: *** cfs_fail_loc=e003, val=100*** [ 1829.828402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1829.844651] LNet: *** cfs_fail_loc=e003, val=100*** [ 1829.849486] LNet: Skipped 2 previous similar messages [ 1829.856805] Ping reply received with mapping: [ 1829.858956] found metadata reply NI[192.168.204.12@tcp] [ 1829.861580] 0c cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1829.864621] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1829.867204] 5a 5a 5a 5a 5a 5a 5a 5a [ 1829.982319] LNet: 101373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1829.986974] LNetError: 101373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.015483] LNet: Removed LNI 192.168.204.12@tcp [ 1831.536418] Key type .llcrypt unregistered [ 1831.538567] Key type ._llcrypt unregistered [ 1840.014265] Key type ._llcrypt registered [ 1840.015744] Key type .llcrypt registered [ 1841.128640] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 22:32:54 (1788834774) [ 1841.599668] Key type .llcrypt unregistered [ 1841.603229] Key type ._llcrypt unregistered [ 1847.862403] Key type ._llcrypt registered [ 1847.865557] Key type .llcrypt registered [ 1848.097207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1848.132037] alg: No test for adler32 (adler32-zlib) [ 1849.309281] LNetError: 102410:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1849.911465] Key type .llcrypt unregistered [ 1849.914272] Key type ._llcrypt unregistered [ 1855.323899] Key type ._llcrypt registered [ 1855.327487] Key type .llcrypt registered [ 1856.962978] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 22:33:09 (1788834789) [ 1857.518377] Key type .llcrypt unregistered [ 1857.520475] Key type ._llcrypt unregistered [ 1864.152629] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1864.159251] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1866.584412] Key type ._llcrypt registered [ 1866.590340] Key type .llcrypt registered [ 1866.818322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1866.837377] alg: No test for adler32 (adler32-zlib) [ 1867.742732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1867.777437] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1876.286498] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 22:33:29 (1788834809) [ 1876.445985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1876.483597] LNet: 104182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1876.509954] LNetError: 104182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1876.526448] LNet: Removed LNI 10.1.2.3@tcp [ 1876.542689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1877.107458] Key type .llcrypt unregistered [ 1877.109822] Key type ._llcrypt unregistered [ 1881.013581] Key type ._llcrypt registered [ 1881.015050] Key type .llcrypt registered [ 1881.121496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1881.144170] alg: No test for adler32 (adler32-zlib) [ 1882.111087] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1882.175735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1882.374475] LNet: 104468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.388037] LNetError: 104468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.412741] LNet: Removed LNI 192.168.204.12@tcp [ 1882.950842] Key type .llcrypt unregistered [ 1882.953788] Key type ._llcrypt unregistered [ 1890.899143] Key type ._llcrypt registered [ 1890.900820] Key type .llcrypt registered [ 1891.858023] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 22:33:45 (1788834825) [ 1892.442309] Key type .llcrypt unregistered [ 1892.448146] Key type ._llcrypt unregistered [ 1898.375684] Key type ._llcrypt registered [ 1898.378669] Key type .llcrypt registered [ 1898.543444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1898.579537] alg: No test for adler32 (adler32-zlib) [ 1906.442144] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 22:33:59 (1788834839) [ 1906.759761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1906.796991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1906.868171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1906.913171] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1907.482397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1907.530634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1907.562865] LNet: Removed LNI 192.168.204.12@tcp [ 1907.869262] LNet: 106205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.463048] Key type .llcrypt unregistered [ 1908.471361] Key type ._llcrypt unregistered [ 1922.115595] Key type ._llcrypt registered [ 1922.122272] Key type .llcrypt registered [ 1923.692879] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 22:34:16 (1788834856) [ 1924.102084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1924.119654] alg: No test for adler32 (adler32-zlib) [ 1925.080462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1925.164307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1925.185289] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1926.034524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1926.083754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.12@tcp [ 1926.108700] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1926.112558] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1926.115279] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1926, 0, 0 [ 1926.128992] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1926.134701] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1926.141651] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1926.149621] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1926, 0, 0 [ 1926.154642] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1926.159443] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1926.164580] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1926.168222] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1926, 0, 1976 [ 1926.177543] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1926.184071] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1926.186955] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1926.194668] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1926, 0, 1976 [ 1926.203385] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1932.495530] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 22:34:25 (1788834865) [ 1933.110395] LNet: 107742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.122073] LNetError: 107742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1933.147260] LNet: Removed LNI 192.168.204.12@tcp [ 1933.694557] Key type .llcrypt unregistered [ 1933.697605] Key type ._llcrypt unregistered [ 1940.900800] Key type ._llcrypt registered [ 1940.910923] Key type .llcrypt registered [ 1941.265442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1941.306991] alg: No test for adler32 (adler32-zlib) [ 1942.286705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1942.356774] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1950.974140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1960.163614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1960.250821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1960.303955] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1961.285196] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1961.512486] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1961.527966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1961.539513] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1963.943208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1963.985863] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1974.893660] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 22:35:07 (1788834907) [ 1975.083419] LNet: 109238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1975.098262] LNetError: 109238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.111748] LNet: Removed LNI 192.168.204.12@tcp [ 1976.161089] LNet: Removed LNI 192.168.204.12@tcp1 [ 1976.164473] LNetError: 109238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1976.174222] LNetError: 109238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1976.197506] LNet: Removed LNI 10.1.2.3@tcp99 [ 1976.952472] Key type .llcrypt unregistered [ 1976.957131] Key type ._llcrypt unregistered [ 1983.321833] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1983.331692] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1985.908360] Key type ._llcrypt registered [ 1985.914722] Key type .llcrypt registered [ 1986.142615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1986.159757] alg: No test for adler32 (adler32-zlib) [ 1987.064349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.121488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1987.154105] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1987.944929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1987.969691] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1990.143641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1990.194238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1990.238314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1990.258627] LNetError: 109985:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1990.285223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1990.321948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 1990.606075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1990.671465] LNet: 110005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.680356] LNetError: 110005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1990.686623] LNet: Removed LNI 10.1.2.3@tcp [ 1991.717869] LNet: Removed LNI 192.168.204.12@tcp [ 1992.259722] Key type .llcrypt unregistered [ 1992.261317] Key type ._llcrypt unregistered [ 2002.275931] Key type ._llcrypt registered [ 2002.282726] Key type .llcrypt registered [ 2003.380640] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 22:35:36 (1788834936) [ 2003.708610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2003.724749] alg: No test for adler32 (adler32-zlib) [ 2004.617166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2004.674196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2004.697941] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2005.349282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2005.408566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg412-client@tcp [ 2010.330908] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 22:35:43 (1788834943) [ 2010.538718] LNet: 111534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2010.547731] LNetError: 111534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2010.574727] LNet: Removed LNI 192.168.204.12@tcp [ 2011.108175] Key type .llcrypt unregistered [ 2011.110449] Key type ._llcrypt unregistered [ 2016.675205] Key type ._llcrypt registered [ 2016.679249] Key type .llcrypt registered [ 2016.887955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2016.933888] alg: No test for adler32 (adler32-zlib) [ 2017.854039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2017.901340] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2017.932409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2023.197017] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 22:35:56 (1788834956) [ 2024.548025] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2025.853944] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 22:35:59 (1788834959) [ 2027.233823] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2028.703723] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 22:36:01 (1788834961) [ 2028.796771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2028.818149] LNet: 112945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.824943] LNetError: 112945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.841330] LNet: Removed LNI 192.168.204.12@tcp [ 2028.869633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2028.988317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2029.012592] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2029.592620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2029.646865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2035.223118] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 22:36:08 (1788834968) [ 2035.291321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2035.310538] LNet: 113656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.316746] LNetError: 113656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2035.345924] LNet: Removed LNI 192.168.204.12@tcp [ 2035.375327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2035.426309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2035.465909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2035.992312] Key type .llcrypt unregistered [ 2035.994502] Key type ._llcrypt unregistered [ 2045.414344] Key type ._llcrypt registered [ 2045.417540] Key type .llcrypt registered [ 2046.509390] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 22:36:19 (1788834979) [ 2046.835271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2046.854220] alg: No test for adler32 (adler32-zlib) [ 2047.773468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2047.833516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2047.857751] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2048.413439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2048.464830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2054.068678] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 22:36:27 (1788834987) [ 2054.212350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2054.254427] LNet: 115214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2054.270233] LNetError: 115214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2054.295424] LNet: Removed LNI 192.168.204.12@tcp [ 2054.312152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2054.358683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2054.398234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2059.929559] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 22:36:32 (1788834992) [ 2060.017381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2060.052793] LNet: 115789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.086879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2060.184987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2060.586792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2060.622573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2060.692784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2060.992813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2061.030929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2061.109815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2061.123203] LNet: 115828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.140229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.226206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2061.523992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2061.547829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2061.597553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2061.853661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2061.881039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2067.240833] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 22:36:40 (1788835000) [ 2067.418972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2067.444585] LNet: 116428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2067.460252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2067.515643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2067.538632] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2068.115182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2068.134882] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2068.835753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2068.881872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2069.051444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2069.067938] LNetError: 116707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2069.075302] LNet: Removed LNI 192.168.204.12@tcp [ 2069.087672] LNet: Removed LNI 192.168.204.12@tcp2 [ 2069.108680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2069.163524] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2069.172814] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2069.176029] LNet: 116713:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2069.675933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2069.863504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2070.943369] LNet: Removed LNI 192.168.204.12@tcp [ 2070.947908] LNetError: 116750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2070.958279] LNetError: 116750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2070.978878] LNet: Removed LNI 192.168.204.12@tcp2 [ 2071.020555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2071.085127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2071.113879] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2071.808746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2071.828257] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2072.570543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2072.628597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2072.694670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2072.920772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2072.939170] LNet: 117035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2072.945993] LNet: 117035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2072.962724] LNetError: 117035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2072.969531] LNet: Removed LNI 192.168.204.12@tcp [ 2074.023863] LNet: Removed LNI 192.168.204.12@tcp2 [ 2074.049546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2074.101523] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2074.107745] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2074.112175] LNet: 117041:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2074.710981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2074.871382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2074.901232] LNet: Removed LNI 192.168.204.12@tcp [ 2075.949691] LNet: Removed LNI 192.168.204.12@tcp2 [ 2075.967180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.010704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2076.036359] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2076.660277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2076.689131] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2077.408739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2077.470884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2077.683183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2077.719155] LNetError: 117357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2077.725143] LNetError: 117357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2077.730867] LNet: Removed LNI 192.168.204.12@tcp [ 2077.747863] LNet: Removed LNI 192.168.204.12@tcp2 [ 2077.768543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2077.814525] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2077.821107] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2077.827242] LNet: 117363:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2078.163665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2078.359081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2079.391243] LNet: Removed LNI 192.168.204.12@tcp [ 2079.403167] LNet: Removed LNI 192.168.204.12@tcp2 [ 2079.445138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.534724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2079.558769] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2080.343085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2080.367626] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2081.223380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2081.258888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2081.320611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2081.601320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2081.633513] LNet: 117685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2081.639911] LNet: 117685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2082.655168] LNet: Removed LNI 192.168.204.12@tcp [ 2083.703123] LNet: Removed LNI 192.168.204.12@tcp2 [ 2083.729630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2083.792428] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2083.798742] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2083.803675] LNet: 117691:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2084.537860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2084.801737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2084.824951] LNet: Removed LNI 192.168.204.12@tcp [ 2085.862891] LNet: Removed LNI 192.168.204.12@tcp2 [ 2085.912585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2085.980623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2086.021843] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2086.776915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2086.797232] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2087.693599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2087.729939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2087.779516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2088.168541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2088.185608] LNetError: 118025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2088.190885] LNetError: 118025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2089.247150] LNet: Removed LNI 192.168.204.12@tcp [ 2089.262274] LNet: Removed LNI 192.168.204.12@tcp2 [ 2089.290575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2089.363470] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2089.370876] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2089.382306] LNet: 118031:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2090.011360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2090.416884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2091.487163] LNet: Removed LNI 192.168.204.12@tcp [ 2091.498698] LNet: Removed LNI 192.168.204.12@tcp2 [ 2091.535349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2091.629747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2091.666097] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2092.495239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2092.529371] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2093.471769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2093.534441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2093.585812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2093.670809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2094.101350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2095.199187] LNet: Removed LNI 192.168.204.12@tcp [ 2096.230498] LNet: Removed LNI 192.168.204.12@tcp2 [ 2096.280355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.359641] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2096.364879] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2096.905149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2103.364063] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2104.738326] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 22:37:17 (1788835037) [ 2104.970585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2104.991318] LNet: 119150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.996415] LNet: 119150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2105.002809] LNetError: 119150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2105.008396] LNetError: 119150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2105.015212] LNet: Removed LNI 192.168.204.12@tcp [ 2106.088158] LNet: Removed LNI 192.168.204.12@tcp2 [ 2106.136211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.383312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2106.411625] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2107.005697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2107.027360] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2107.932275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2107.960289] LNet: Removed LNI 10.1.2.3@tcp [ 2107.973429] LNet: Removed LNI 192.168.204.12@tcp [ 2107.995455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.059376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2108.084429] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2108.088660] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2108.479063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2108.501864] LNet: Removed LNI 192.168.204.12@tcp [ 2108.510183] LNet: Removed LNI 10.1.2.3@tcp [ 2114.410972] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 22:37:27 (1788835047) [ 2114.546116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2114.589947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2114.645886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2114.673323] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2115.469887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2115.534655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2115.605216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2115.654538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2115.705715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2115.766946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2115.802244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2115.850825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2115.891533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2115.971227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2116.024226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2116.225642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2121.503037] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 22:37:34 (1788835054) [ 2121.610565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2122.663217] LNet: Removed LNI 192.168.204.12@tcp [ 2122.678677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.773840] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2122.874094] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2122.876225] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2122.879122] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2125.125441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2125.149542] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2125.737857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2125.756407] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2127.261308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2127.297559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2127.365203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.12@tcp -m PUT -e local_timeout -r 1 [ 2127.413837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2127.486477] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2127.493660] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2127.680667] LNetError: 120823:0:(lib-move.c:4801:lnet_parse()) 192.168.204.12@tcp, src 192.168.204.12@tcp: Dropping PUT (error -108 looking up sender) [ 2128.739797] LNet: Removed LNI 10.1.2.3@tcp [ 2129.768499] LNet: Removed LNI 192.168.204.12@tcp [ 2130.117544] Key type .llcrypt unregistered [ 2130.120317] Key type ._llcrypt unregistered [ 2140.810369] Key type ._llcrypt registered [ 2140.812513] Key type .llcrypt registered [ 2142.124482] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 22:37:55 (1788835075) [ 2142.489920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2142.518454] alg: No test for adler32 (adler32-zlib) [ 2143.434837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.797862] LNet: 122130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.449447] Key type .llcrypt unregistered [ 2144.454971] Key type ._llcrypt unregistered [ 2150.454360] Key type ._llcrypt registered [ 2150.458503] Key type .llcrypt registered [ 2150.607409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2150.630288] alg: No test for adler32 (adler32-zlib) [ 2151.552795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2151.572526] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2158.714948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 2158.943558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2159.634866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.112@tcp [ 2159.782146] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.112@tcp [ 2159.879518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2159.921458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2159.956304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2159.992327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2160.027353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2160.070709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2160.115794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2160.155170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2160.182030] LNet: 122576:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2160.200489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2160.241092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2160.265110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2160.296757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2160.329541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2160.366375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2160.914559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2161.138105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2161.166363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2161.214987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2161.244286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2161.274165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2161.304410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2161.344498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2161.373244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2161.388610] LNet: 122693:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2161.417813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2161.469759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2161.507456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2161.543459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2161.573371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2161.609943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2162.173280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.112@tcp [ 2162.387875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2162.415615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2162.437892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2162.468532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2162.507862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2162.539653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2162.567995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2162.594659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2162.620785] LNet: 122811:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2162.641290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2162.669851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2162.701590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2162.739731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2162.775389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2162.804595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2163.331915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2163.523268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2163.544950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2163.571594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2163.601960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2163.627831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2163.651419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2163.672589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2163.708193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2163.736745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2163.762535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2163.796674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2163.847957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2163.878509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2163.910564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2164.239996] LNet: 122974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2164.246411] LNetError: 122974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2164.275157] LNet: Removed LNI 192.168.204.12@tcp [ 2164.632933] Key type .llcrypt unregistered [ 2164.635705] Key type ._llcrypt unregistered [ 2179.712762] Key type ._llcrypt registered [ 2179.714914] Key type .llcrypt registered [ 2180.877919] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 22:38:33 (1788835113) [ 2181.486211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2181.515845] alg: No test for adler32 (adler32-zlib) [ 2182.419639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.771208] LNet: 123857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2183.453523] Key type .llcrypt unregistered [ 2183.459250] Key type ._llcrypt unregistered [ 2189.588242] Key type ._llcrypt registered [ 2189.591700] Key type .llcrypt registered [ 2189.851353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2189.901655] alg: No test for adler32 (adler32-zlib) [ 2190.904293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2190.938501] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2198.396983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 2206.814445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2206.853096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2206.867762] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 2207.581861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2208.122903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.112@tcp [ 2208.352733] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.112@tcp [ 2208.442972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2208.471479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2208.503220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2208.535559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2208.580710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2208.642619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2208.670860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2208.702918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2208.719173] LNet: 124506:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2208.737528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2208.763725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2208.792881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2208.815247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2208.843507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2208.865203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2209.446747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2209.660740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2209.688964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2209.725563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2209.761411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2209.793396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2209.821901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2209.849838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2209.882659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2209.907061] LNet: 124624:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2209.935683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2209.984969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2210.028605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2210.060976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2210.097650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2210.140889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2210.694274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.112@tcp [ 2210.989796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2211.022927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2211.042169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2211.070303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2211.110537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2211.148987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2211.193085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2211.225582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2211.251427] LNet: 124741:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2211.280918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2211.308557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2211.332430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2211.357538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2211.382156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2211.415666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2212.093809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2212.350717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2212.384450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2212.412531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2212.456629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2212.484250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2212.512937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2212.537334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2212.562549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2212.608982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2212.640619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2212.685242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2212.728371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2212.787237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2212.853809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2218.500854] LNet: 124939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.505541] LNetError: 124939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2218.516664] LNet: Removed LNI 192.168.204.12@tcp [ 2218.534447] LNet: Removed LNI 192.168.204.12@tcp1 [ 2218.998733] Key type .llcrypt unregistered [ 2219.000336] Key type ._llcrypt unregistered [ 2236.536485] Key type ._llcrypt registered [ 2236.540777] Key type .llcrypt registered [ 2237.748486] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 22:39:30 (1788835170) [ 2238.225304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2238.240916] alg: No test for adler32 (adler32-zlib) [ 2239.104379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.159446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2239.175935] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2239.716839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2239.736837] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2240.615692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2240.677335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2241.695542] LNet: Removed LNI 192.168.204.12@tcp [ 2247.721387] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 22:39:40 (1788835180) [ 2249.174984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2250.935414] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 22:39:43 (1788835183) [ 2258.028234] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 22:39:51 (1788835191) [ 2265.402753] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 22:39:58 (1788835198) [ 2272.208679] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 22:40:05 (1788835205) [ 2278.585495] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 22:40:11 (1788835211) [ 2284.668149] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 22:40:17 (1788835217) [ 2291.285717] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 22:40:24 (1788835224) [ 2298.149425] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 22:40:31 (1788835231) [ 2304.402893] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 22:40:37 (1788835237) [ 2309.979807] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 22:40:43 (1788835243) [ 2315.637336] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 22:40:48 (1788835248) [ 2322.982131] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 22:40:55 (1788835255) [ 2329.062667] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 22:41:02 (1788835262) [ 2335.670753] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 22:41:08 (1788835268) [ 2343.081352] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 22:41:15 (1788835275) [ 2350.632756] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 22:41:23 (1788835283) [ 2358.288336] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 22:41:30 (1788835290) [ 2366.631480] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 22:41:39 (1788835299) [ 2366.818376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2366.857478] LNet: 136627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2366.874874] LNetError: 136627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2366.914316] LNet: Removed LNI 192.168.204.12@tcp2 [ 2366.951446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2367.157721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2367.218045] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2368.559167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2368.596774] LNet: 136769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2368.611689] LNetError: 136769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2368.625459] LNet: Removed LNI 192.168.204.12@tcp [ 2368.665306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2368.743842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2368.780642] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2375.600376] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 22:41:48 (1788835308) [ 2382.824780] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 22:41:55 (1788835315) [ 2388.640182] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 22:42:01 (1788835321) [ 2394.953103] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 22:42:08 (1788835328) [ 2400.687179] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 22:42:13 (1788835333) [ 2406.899440] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 22:42:20 (1788835340) [ 2413.720616] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 22:42:26 (1788835346) [ 2420.831780] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 22:42:34 (1788835354) [ 2426.916989] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 22:42:39 (1788835359) [ 2433.869870] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 22:42:46 (1788835366) [ 2441.211695] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 22:42:53 (1788835373) [ 2449.217119] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 22:43:02 (1788835382) [ 2456.591752] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2219 sec ========== 22:43:09 (1788835389) [ 2457.015179] LNet: 144233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2457.020907] LNetError: 144233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2457.036359] LNet: Removed LNI 192.168.204.12@tcp [ 2457.578815] Key type .llcrypt unregistered [ 2457.585894] Key type ._llcrypt unregistered [ 2470.572503] Key type ._llcrypt registered [ 2470.602935] Key type .llcrypt registered [ 2471.265948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2471.281405] alg: No test for adler32 (adler32-zlib) [ 2472.364474] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 2472.645644] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2474.431430] Key type lgssc registered [ 2475.644119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2538.221775] Lustre: Mounted lustre-client - version 2.17.57_106_g45f4cc0 [ 2543.430330] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2550.442507] Lustre: Unmounted lustre-client [ 2577.617966] Key type lgssc unregistered [ 2577.879961] LNet: 147179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2577.895918] LNetError: 147179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2577.916176] LNet: Removed LNI 192.168.204.12@tcp [ 2578.675147] Key type .llcrypt unregistered [ 2578.679956] Key type ._llcrypt unregistered