[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489366151 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003037] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.007910] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.011116] LSM: Security Framework initializing [ 0.012045] Yama: becoming mindful. [ 0.013029] SELinux: Initializing. [ 0.014057] *** VALIDATE selinux *** [ 0.022056] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027210] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030103] *** VALIDATE tmpfs *** [ 0.032433] *** VALIDATE proc *** [ 0.033217] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035250] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036144] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038024] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.041734] debug: unmapping init [mem 0xffffffffaf459000-0xffffffffaf460fff] [ 0.043155] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044707] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045019] ... version: 2 [ 0.046009] ... bit width: 48 [ 0.047008] ... generic registers: 4 [ 0.048010] ... value mask: 0000ffffffffffff [ 0.049011] ... max period: 00007fffffffffff [ 0.050010] ... fixed-purpose events: 3 [ 0.051009] ... event mask: 000000070000000f [ 0.052300] rcu: Hierarchical SRCU implementation. [ 0.054460] smp: Bringing up secondary CPUs ... [ 0.055539] x86: Booting SMP configuration: [ 0.056021] .... node #0, CPUs: #1 #2 #3 [ 0.062016] smp: Brought up 1 node, 4 CPUs [ 0.064016] smpboot: Max logical packages: 1 [ 0.065013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135740] node 0 deferred pages initialised in 67ms [ 0.140012] devtmpfs: initialized [ 0.141202] x86/mm: Memory block size: 128MB [ 0.143741] gcov: version magic: 0x41383552 [ 0.144577] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154463] pinctrl core: initialized pinctrl subsystem [ 0.157250] [ 0.157956] ************************************************************* [ 0.160012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163014] ** ** [ 0.165013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168013] ** ** [ 0.172014] ** This means that this kernel is built to expose internal ** [ 0.174013] ** IOMMU data structures, which may compromise security on ** [ 0.177015] ** your system. ** [ 0.180014] ** ** [ 0.182019] ** If you see this message and you are not debugging the ** [ 0.185047] ** kernel, report this immediately to your vendor! ** [ 0.188011] ** ** [ 0.190011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192012] ************************************************************* [ 0.198805] NET: Registered protocol family 16 [ 0.200650] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.204085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206129] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211122] cpuidle: using governor menu [ 0.212807] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.214471] PCI: Using configuration type 1 for base access [ 0.217128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.226076] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.228094] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.232104] cryptd: max_cpu_qlen set to 1000 [ 0.236274] ACPI: Added _OSI(Module Device) [ 0.238010] ACPI: Added _OSI(Processor Device) [ 0.240013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.242015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.247897] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.255630] ACPI: Interpreter enabled [ 0.257056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.258009] ACPI: Using IOAPIC for interrupt routing [ 0.260095] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.263312] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.274945] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.277081] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.279014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.283058] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287471] acpiphp: Slot [2] registered [ 0.288143] acpiphp: Slot [5] registered [ 0.290097] acpiphp: Slot [6] registered [ 0.292110] acpiphp: Slot [3] registered [ 0.294092] acpiphp: Slot [4] registered [ 0.295089] acpiphp: Slot [7] registered [ 0.296116] acpiphp: Slot [8] registered [ 0.298088] acpiphp: Slot [9] registered [ 0.300103] acpiphp: Slot [10] registered [ 0.302123] acpiphp: Slot [11] registered [ 0.303236] acpiphp: Slot [12] registered [ 0.306177] acpiphp: Slot [13] registered [ 0.308114] acpiphp: Slot [14] registered [ 0.310112] acpiphp: Slot [15] registered [ 0.313341] acpiphp: Slot [16] registered [ 0.315128] acpiphp: Slot [17] registered [ 0.317099] acpiphp: Slot [18] registered [ 0.319096] acpiphp: Slot [19] registered [ 0.321100] acpiphp: Slot [20] registered [ 0.323260] acpiphp: Slot [21] registered [ 0.325105] acpiphp: Slot [22] registered [ 0.327321] acpiphp: Slot [23] registered [ 0.329193] acpiphp: Slot [24] registered [ 0.332114] acpiphp: Slot [25] registered [ 0.334226] acpiphp: Slot [26] registered [ 0.335095] acpiphp: Slot [27] registered [ 0.337165] acpiphp: Slot [28] registered [ 0.338082] acpiphp: Slot [29] registered [ 0.340109] acpiphp: Slot [30] registered [ 0.341088] acpiphp: Slot [31] registered [ 0.343074] PCI host bridge to bus 0000:00 [ 0.345033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.348021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.350066] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.354029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.357027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.361026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.363170] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.365028] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.368357] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.379013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.386023] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.388018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.391017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.394018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.396500] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.400860] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.403041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.407000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.412012] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.423014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.428737] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.435698] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.447015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.457025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.480027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.494028] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.499013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.504011] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.518021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.528375] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.531490] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.535373] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.537300] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.540232] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.544158] iommu: Default domain type: Passthrough [ 0.547099] SCSI subsystem initialized [ 0.550245] ACPI: bus type USB registered [ 0.553234] usbcore: registered new interface driver usbfs [ 0.556104] usbcore: registered new interface driver hub [ 0.559471] usbcore: registered new device driver usb [ 0.563244] pps_core: LinuxPPS API ver. 1 registered [ 0.566015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.570287] PTP clock support registered [ 0.573150] EDAC MC: Ver: 3.0.0 [ 0.575402] PCI: Using ACPI for IRQ routing [ 0.578570] NetLabel: Initializing [ 0.580015] NetLabel: domain hash size = 128 [ 0.581009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.582083] NetLabel: unlabeled traffic allowed by default [ 0.584106] vgaarb: loaded [ 0.585287] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.588018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.597343] clocksource: Switched to clocksource kvm-clock [ 0.719632] VFS: Disk quotas dquot_6.6.0 [ 0.720776] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.722487] *** VALIDATE ramfs *** [ 0.723412] *** VALIDATE hugetlbfs *** [ 0.724787] pnp: PnP ACPI init [ 0.727242] pnp: PnP ACPI: found 6 devices [ 0.743048] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.748368] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.750896] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.753766] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.757487] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.761252] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.765467] NET: Registered protocol family 2 [ 0.768513] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.776504] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.781739] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.786298] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.789976] TCP: Hash tables configured (established 65536 bind 65536) [ 0.792897] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.796325] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.799684] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.803163] NET: Registered protocol family 1 [ 0.806266] RPC: Registered named UNIX socket transport module. [ 0.808979] RPC: Registered udp transport module. [ 0.811343] RPC: Registered tcp transport module. [ 0.813966] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.816809] NET: Registered protocol family 44 [ 0.818600] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.821065] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.823642] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.826612] PCI: CLS 0 bytes, default 64 [ 0.829833] Unpacking initramfs... [ 2.316583] debug: unmapping init [mem 0xffff89b57cc64000-0xffff89b57ffcffff] [ 2.323091] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.325670] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.329787] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.804114] Initialise system trusted keyrings [ 2.807138] Key type blacklist registered [ 2.809237] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.817936] zbud: loaded [ 2.820640] *** VALIDATE nfs *** [ 2.821763] *** VALIDATE nfs4 *** [ 2.823117] pstore: using deflate compression [ 2.826519] Platform Keyring initialized [ 2.938884] NET: Registered protocol family 38 [ 2.940920] Key type asymmetric registered [ 2.942487] Asymmetric key parser 'x509' registered [ 2.944480] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.947938] io scheduler mq-deadline registered [ 2.950115] io scheduler kyber registered [ 2.952385] io scheduler bfq registered [ 2.962273] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.966264] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.971191] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.975152] ACPI: Power Button [PWRF] [ 2.982918] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.991277] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.008241] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.036195] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.065049] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.069789] Non-volatile memory driver v1.3 [ 3.071526] Linux agpgart interface v0.103 [ 3.102260] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.105028] vda: detected capacity change from 0 to 76775424 [ 3.126080] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.130872] vdb: detected capacity change from 0 to 1073741824 [ 3.143202] libphy: Fixed MDIO Bus: probed [ 3.153265] usbcore: registered new interface driver usbserial_generic [ 3.158228] usbserial: USB Serial support registered for generic [ 3.160119] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.166733] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.168553] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.173266] mousedev: PS/2 mouse device common for all mice [ 3.177668] rtc_cmos 00:05: RTC can wake from S4 [ 3.182725] rtc_cmos 00:05: registered as rtc0 [ 3.186584] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.189869] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.189893] intel_pstate: CPU model not supported [ 3.196885] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.201582] hid: raw HID events driver (C) Jiri Kosina [ 3.206532] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.207243] usbcore: registered new interface driver usbhid [ 3.212215] usbhid: USB HID core driver [ 3.214212] drop_monitor: Initializing network drop monitor service [ 3.217570] Initializing XFRM netlink socket [ 3.220583] NET: Registered protocol family 10 [ 3.224759] Segment Routing with IPv6 [ 3.226460] NET: Registered protocol family 17 [ 3.228690] mpls_gso: MPLS GSO support [ 3.236971] RAS: Correctable Errors collector initialized. [ 3.238838] AVX version of gcm_enc/dec engaged. [ 3.241636] AES CTR mode by8 optimization enabled [ 3.335422] sched_clock: Marking stable (3335388967, 0)->(4209779912, -874390945) [ 3.339381] registered taskstats version 1 [ 3.342653] Loading compiled-in X.509 certificates [ 3.345627] zswap: loaded using pool lzo/zbud [ 3.373760] Key type big_key registered [ 3.386341] Key type encrypted registered [ 3.388216] ima: No TPM chip found, activating TPM-bypass! [ 3.390867] ima: Allocated hash algorithm: sha1 [ 3.393794] ima: No architecture policies found [ 3.396241] evm: Initialising EVM extended attributes: [ 3.398761] evm: security.selinux [ 3.400539] evm: security.ima [ 3.401984] evm: security.capability [ 3.403827] evm: HMAC attrs: 0x1 [ 3.406896] rtc_cmos 00:05: setting system clock to 2026-09-07 15:52:54 UTC (1788796374) [ 3.414986] debug: unmapping init [mem 0xffffffffb0403000-0xffffffffb05fffff] [ 3.419446] debug: unmapping init [mem 0xffffffffaf182000-0xffffffffaf458fff] [ 3.429073] Write protecting the kernel read-only data: 28672k [ 3.432656] debug: unmapping init [mem 0xffffffffad803000-0xffffffffad9fffff] [ 3.435601] debug: unmapping init [mem 0xffffffffae114000-0xffffffffae1fffff] [ 3.473557] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.484415] systemd[1]: Detected virtualization kvm. [ 3.486216] systemd[1]: Detected architecture x86-64. [ 3.487942] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.516855] systemd[1]: No hostname configured. [ 3.520417] systemd[1]: Set hostname to . [ 3.523726] random: systemd: uninitialized urandom read (16 bytes read) [ 3.528578] systemd[1]: Initializing machine ID from random generator. [ 3.667787] random: systemd: uninitialized urandom read (16 bytes read) [ 3.671794] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.682255] random: systemd: uninitialized urandom read (16 bytes read) [ 3.685211] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.690892] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. Starting dracut cmdline hook... [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.410241] device-mapper: uevent: version 1.0.3 [ 4.412321] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.170407] virtio_net virtio0 ens2: renamed from eth0 [ 5.218973] scsi host0: ata_piix [ 5.234670] scsi host1: ata_piix [ 5.237401] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.240312] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.851857] dracut-initqueue[585]: RTNETLINK answers: File exists[ 9.855223] random: fast init done [ 9.911972] random: crng init done [ 9.913584] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.585971] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.059832] printk: systemd: 24 output lines suppressed due to ratelimiting [ 12.376145] SELinux: Disabled at runtime. [ 12.436476] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.445384] systemd[1]: Detected virtualization kvm. [ 12.447245] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.078872] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.082424] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.090794] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.096747] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.101209] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.109370] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.113239] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ 13.161085] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.752731] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.124927] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.209220] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.350778] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.379500] EDAC sbridge: Ver: 1.1.2 [ 15.873136] Key type dns_resolver registered [ 16.209387] NFS: Registering the id_resolver key type [ 16.211539] Key type id_resolver registered [ 16.213373] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg649-client login: [ 48.667268] libcfs: loading out-of-tree module taints kernel. [ 48.916094] Key type ._llcrypt registered [ 48.918923] Key type .llcrypt registered [ 49.380339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 49.397808] alg: No test for adler32 (adler32-zlib) [ 50.956804] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 51.784555] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 53.680197] Key type lgssc registered [ 55.284245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 115.328004] hrtimer: interrupt took 1997274 ns [ 184.608966] Lustre: Mounted lustre-client - version 2.17.58_39_g569731d [ 189.436527] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 206.649971] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing check_logdir /tmp/testlogs/ [ 210.399387] Lustre: lustre-OST0000-osc-ffff89b5c65b0800: disconnect after 24s idle [ 211.212642] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing yml_node [ 214.657133] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 216.836435] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 219.308929] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 221.038891] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Sep 7 11:56:30 EDT 2026 [ 238.427890] Lustre: Unmounted lustre-client [ 263.655243] Key type lgssc unregistered [ 263.940940] LNet: 8430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 263.953968] LNetError: 8430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 263.978484] LNet: Removed LNI 192.168.206.49@tcp [ 264.603154] Key type .llcrypt unregistered [ 264.606351] Key type ._llcrypt unregistered [ 284.587821] Key type ._llcrypt registered [ 284.589893] Key type .llcrypt registered [ 284.741377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 284.751704] alg: No test for adler32 (adler32-zlib) [ 285.656626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 285.722014] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 285.749063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 288.871892] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 289.213962] LNet: 9308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 289.220545] LNetError: 9308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 289.235908] LNet: Removed LNI 192.168.206.49@tcp [ 289.743786] Key type .llcrypt unregistered [ 289.748465] Key type ._llcrypt unregistered [ 294.199033] Key type ._llcrypt registered [ 294.201399] Key type .llcrypt registered [ 295.209664] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 11:57:45 (1788796665) [ 295.447914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 295.458583] alg: No test for adler32 (adler32-zlib) [ 296.357837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.424643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.444317] LNet: 9715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 296.459445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.507600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 296.520650] LNet: 9722:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 301.758555] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 11:57:51 (1788796671) [ 301.949078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.977560] LNet: 10299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.001291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.061818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 307.471159] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 11:57:57 (1788796677) [ 307.594242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.637975] LNet: 10881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.682638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.758628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 312.817979] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 11:58:02 (1788796682) [ 312.885519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 312.906185] LNet: 11463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.926553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.010283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 317.884219] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 11:58:07 (1788796687) [ 317.945363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.968985] LNet: 12044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.997492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.067811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 318.462799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 322.841429] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 11:58:13 (1788796693) [ 323.120729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.153781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.230510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 323.649898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 323.973164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 328.659960] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 11:58:18 (1788796698) [ 328.761523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.804395] LNet: 13232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.811906] LNet: 13232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 328.840629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.939351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 334.027380] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 11:58:24 (1788796704) [ 334.098224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.145435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.213195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 334.240464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 334.577686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 334.633903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 335.031958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 335.055843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 335.409451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 335.450975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 335.793778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 335.821389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 336.146875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 336.179270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 336.484969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 336.519813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 336.888326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 336.913598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 337.255559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 337.288300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 337.701430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 337.732560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 337.979470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 338.005099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 338.361068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 338.385766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 338.719597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 338.749775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 343.609860] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 11:58:33 (1788796713) [ 343.680230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.733970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.830875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 343.862684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 348.398440] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 11:58:38 (1788796718) [ 348.488656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.509767] LNet: 15088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.514562] LNet: 15088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 348.530512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.593230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 348.623447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 353.193434] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 11:58:43 (1788796723) [ 353.340958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.365309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.426876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 353.460719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 358.208198] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 11:58:48 (1788796728) [ 358.360486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.405097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.471341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 358.508233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 363.688155] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 11:58:53 (1788796733) [ 363.814301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.844830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.898307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 363.931856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 368.952402] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 11:58:58 (1788796738) [ 369.156909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.203842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.297417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 369.357314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 374.932203] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 11:59:05 (1788796745) [ 375.060288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.104983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.178551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.473542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 380.486431] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 11:59:10 (1788796750) [ 380.582927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.608806] LNet: 18615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.617575] LNet: 18615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 380.646368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.831829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 381.024348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 385.977568] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 11:59:15 (1788796755) [ 386.059423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.107590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.253804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 386.482614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 392.030092] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 11:59:21 (1788796761) [ 392.126421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.168930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.511910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 392.701557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 398.297596] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 11:59:28 (1788796768) [ 398.416123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.458401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.547080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 398.591754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 403.583296] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 11:59:33 (1788796773) [ 403.747937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.784545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.864471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 403.907597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 408.493246] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 11:59:38 (1788796778) [ 408.651633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.682783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.721879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 413.070248] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 11:59:43 (1788796783) [ 413.126760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.158911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.238229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 417.557895] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 11:59:47 (1788796787) [ 417.613942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.670970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.755575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 418.056267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 422.266838] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 11:59:52 (1788796792) [ 422.404842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.435589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.497993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 422.517920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 426.780392] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 11:59:56 (1788796796) [ 426.866177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.888512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.954900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 426.986376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 431.704283] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:00:01 (1788796801) [ 431.793832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.817533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.891681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 431.919661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 437.715370] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:00:07 (1788796807) [ 437.999446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.057645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.183421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 438.251467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 445.181920] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:00:15 (1788796815) [ 445.315790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.354477] LNet: 25806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 445.367210] LNet: 25806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 445.416786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.477201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 445.549940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 445.569954] LNetError: 25820:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 445.625371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 445.671357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 452.397268] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:00:22 (1788796822) [ 452.890919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.911980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.028394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.CI92nEsZ8K [ 453.093243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.CI92nEsZ8K [ 453.182118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.CI92nEsZ8K [ 459.892863] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:00:29 (1788796829) [ 460.074649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.119398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.232292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 460.263501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 460.315874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 467.513844] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 12:00:36 (1788796836) [ 467.861518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.931352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.062363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 468.084377] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 469.514245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 470.793653] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 472.767496] Key type lgssc registered [ 474.002449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 477.465357] LNetError: 27581:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.206.49@tcp99 has no available nets [ 479.958491] Key type lgssc unregistered [ 480.384019] LNetError: 28687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 480.403633] LNet: Removed LNI 192.168.206.49@tcp [ 481.045127] Key type .llcrypt unregistered [ 481.057285] Key type ._llcrypt unregistered [ 487.931432] Key type ._llcrypt registered [ 487.933429] Key type .llcrypt registered [ 489.097524] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:00:58 (1788796858) [ 489.530533] Key type .llcrypt unregistered [ 489.539118] Key type ._llcrypt unregistered [ 495.294463] Key type ._llcrypt registered [ 495.296693] Key type .llcrypt registered [ 495.441314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 495.452656] alg: No test for adler32 (adler32-zlib) [ 496.420843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.478985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 496.509712] LNet: 29610:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 496.518324] LNet: 29610:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 496.527626] LNet: 29610:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 496.551926] LNet: Message forwarding will be enabled in 2s [ 499.679158] LNet: Message forwarding enabled [ 499.992444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 500.012308] LNet: Message forwarding will stop in 2s to 6s [ 503.781225] LNet: Message forwarding disabled [ 504.661815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 504.694740] LNet: 29708:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 504.709821] LNet: 29708:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 504.725415] LNet: 29708:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 504.745300] LNet: Message forwarding will be enabled in 2s [ 504.982012] LNet: 29721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 505.564376] Key type .llcrypt unregistered [ 505.568410] Key type ._llcrypt unregistered [ 514.919331] Key type ._llcrypt registered [ 514.921363] Key type .llcrypt registered [ 516.128259] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:01:25 (1788796885) [ 516.584254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 516.610994] alg: No test for adler32 (adler32-zlib) [ 517.597369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.683575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 523.869922] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:01:33 (1788796893) [ 523.967598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.000469] LNet: 31116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 524.033719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.114181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 524.156256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 524.187655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 524.214965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 524.254605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 524.296222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 524.321716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 524.353344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 524.387765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 524.429977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 524.478690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 524.519333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 524.548426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 524.577778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 524.611345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 524.647095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 524.677592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 524.728542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 524.762051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 524.803228] LNetError: 31164:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 530.112710] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:01:39 (1788796899) [ 530.266973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.290501] LNet: 31731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 530.323836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.421537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 536.911902] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:01:46 (1788796906) [ 538.671742] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg649-server [ 540.572404] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:01:50 (1788796910) [ 540.705615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.743582] LNet: 32659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 540.777807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.849895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 540.872374] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 541.768257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[50-54]@tcp [ 541.790885] LNet: 32799:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 541.834583] LNetError: 32669:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.50@tcp is being used as a gateway but routing feature is not turned on [ 541.858166] LNetError: 32669:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.50@tcp has gone from up to down [ 544.865032] LNetError: 32800:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.53@tcp has gone from up to down [ 544.883677] LNetError: 32800:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 547.758852] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:01:57 (1788796917) [ 549.455416] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg649-server [ 551.095477] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:02:00 (1788796920) [ 551.167577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.192223] LNet: 33728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.201681] LNetError: 33728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 551.221447] LNet: Removed LNI 192.168.206.49@tcp [ 551.252259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.323394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 551.350957] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 551.878584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[50-55/2]@tcp [ 551.894385] LNet: 33852:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 551.900483] LNet: 33852:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 551.921206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[50-55/2]@tcp [ 557.419303] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:02:07 (1788796927) [ 557.558983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.595394] LNet: 34443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 557.602989] LNetError: 34443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 557.627736] LNet: Removed LNI 192.168.206.49@tcp [ 557.657269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.727824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 557.776960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.818051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.876147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 557.933287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.963306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.014385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 558.081891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.124817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.197872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 558.301042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.345446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.395083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 558.455543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.501512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.556811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 558.628841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.659956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.703345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 558.777128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.807846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.854903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 558.941199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.005292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.059233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 559.149559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.196473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.258223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 559.327232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.368965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.426841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 559.503804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.551975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.611525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 565.715455] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:02:15 (1788796935) [ 567.571648] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg649-server [ 569.334445] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:02:18 (1788796938) [ 571.030968] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg649-server [ 572.486421] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:02:22 (1788796942) [ 572.640698] LNet: 35838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 572.649907] LNet: 35838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 573.355239] Key type .llcrypt unregistered [ 573.367318] Key type ._llcrypt unregistered [ 579.382102] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 579.535614] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 579.551119] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 579.561974] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 582.078785] Key type ._llcrypt registered [ 582.081154] Key type .llcrypt registered [ 582.286714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 582.304753] alg: No test for adler32 (adler32-zlib) [ 583.213140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.270522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 583.298444] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 583.964943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 583.987042] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 586.154855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 586.188903] LNet: Removed LNI 10.1.2.3@tcp [ 586.295725] LNet: 36554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.301944] LNetError: 36554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 586.313854] LNet: Removed LNI 192.168.206.49@tcp [ 587.006763] Key type .llcrypt unregistered [ 587.009866] Key type ._llcrypt unregistered [ 599.088267] Key type ._llcrypt registered [ 599.090969] Key type .llcrypt registered [ 600.906725] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:02:50 (1788796970) [ 601.604323] Key type .llcrypt unregistered [ 601.607925] Key type ._llcrypt unregistered [ 610.444850] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 610.664049] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 610.668752] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 610.672180] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 613.468326] Key type ._llcrypt registered [ 613.479899] Key type .llcrypt registered [ 613.749814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 613.773122] alg: No test for adler32 (adler32-zlib) [ 614.826935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.966208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 615.040914] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 616.350617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 616.377649] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 619.554368] LNet: 38039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 619.559772] LNetError: 38039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 619.564465] LNet: Removed LNI 10.1.2.3@tcp [ 619.573503] LNet: Removed LNI 192.168.206.49@tcp [ 620.051879] Key type .llcrypt unregistered [ 620.054976] Key type ._llcrypt unregistered [ 631.971306] Key type ._llcrypt registered [ 631.977821] Key type .llcrypt registered [ 633.505135] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:03:22 (1788797002) [ 633.830430] Key type .llcrypt unregistered [ 633.832487] Key type ._llcrypt unregistered [ 638.859234] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 639.014771] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 639.025063] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 639.029871] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 641.632204] Key type ._llcrypt registered [ 641.636864] Key type .llcrypt registered [ 641.901410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 641.934163] alg: No test for adler32 (adler32-zlib) [ 642.896473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.946994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 642.975689] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 644.692770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 644.712718] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 647.263368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 647.277755] LNet: Removed LNI 10.1.2.3@tcp [ 647.301291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 647.336724] LNet: Removed LNI 10.1.2.31@tcp [ 647.353804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 647.384655] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 648.349850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 648.379343] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 649.227799] LNet: 39952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.233254] LNetError: 39952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 649.239969] LNet: Removed LNI 10.1.2.31@tcp [ 649.265639] LNet: Removed LNI 10.1.2.3@tcp [ 650.179248] Key type .llcrypt unregistered [ 650.181701] Key type ._llcrypt unregistered [ 662.187670] Key type ._llcrypt registered [ 662.189748] Key type .llcrypt registered [ 663.234370] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:03:53 (1788797033) [ 663.637836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 663.653235] alg: No test for adler32 (adler32-zlib) [ 664.561765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.617369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 664.641161] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 665.404809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.434695] LNet: 40903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.446481] LNetError: 40903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.474019] LNet: Removed LNI 192.168.206.49@tcp [ 665.502796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.577331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.49@tcp [ 665.598942] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 666.307187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.49@tcp [ 666.329351] LNet: Removed LNI 192.168.206.49@tcp [ 666.381647] LNet: 41038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.937469] Key type .llcrypt unregistered [ 666.939121] Key type ._llcrypt unregistered [ 678.675894] Key type ._llcrypt registered [ 678.694148] Key type .llcrypt registered [ 679.959446] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:04:09 (1788797049) [ 682.026778] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg649-server [ 683.402868] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:04:13 (1788797053) [ 684.223287] Key type .llcrypt unregistered [ 684.227720] Key type ._llcrypt unregistered [ 690.268335] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 690.470817] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 690.476043] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 690.481220] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 693.184685] Key type ._llcrypt registered [ 693.186284] Key type .llcrypt registered [ 693.364032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 693.392231] alg: No test for adler32 (adler32-zlib) [ 694.340378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.399632] LNet: Added LNI 192.168.206.49@tcp1 [8/256/0/180] [ 695.105529] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 696.922982] LNet: 42869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.928801] LNetError: 42869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.933729] LNet: Removed LNI 192.168.206.49@tcp1 [ 696.945832] LNet: Removed LNI 10.1.2.3@tcp2 [ 697.516466] Key type .llcrypt unregistered [ 697.520230] Key type ._llcrypt unregistered [ 707.821536] Key type ._llcrypt registered [ 707.824429] Key type .llcrypt registered [ 709.352582] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:04:38 (1788797078) [ 709.577497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 709.597881] alg: No test for adler32 (adler32-zlib) [ 710.511310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.573568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 710.592430] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 711.252729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.287108] LNet: 43836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.300436] LNetError: 43836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.314943] LNet: Removed LNI 192.168.206.49@tcp [ 711.343697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.404556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 711.425051] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 716.767058] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:04:46 (1788797086) [ 716.961074] LNet: 44426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.974513] LNetError: 44426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 716.990050] LNet: Removed LNI 192.168.206.49@tcp [ 717.499465] Key type .llcrypt unregistered [ 717.501494] Key type ._llcrypt unregistered [ 723.094462] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 723.336687] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 723.350539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 723.352936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 725.938652] Key type ._llcrypt registered [ 725.940210] Key type .llcrypt registered [ 726.172385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 726.193274] alg: No test for adler32 (adler32-zlib) [ 727.173378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.214890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 727.235736] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 727.839178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 727.856251] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 729.865836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.905143] LNet: 45166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.913690] LNetError: 45166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 729.918594] LNet: Removed LNI 10.1.2.3@tcp [ 729.929578] LNet: Removed LNI 192.168.206.49@tcp [ 729.949862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.993520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 730.010839] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 730.017033] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 730.539943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.556855] LNet: 45193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.562918] LNetError: 45193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 730.567591] LNetError: 45193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 730.573490] LNet: Removed LNI 10.1.2.3@tcp [ 730.586604] LNet: Removed LNI 192.168.206.49@tcp [ 730.607255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.675887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 730.702694] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 730.706768] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 731.176196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 731.201080] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 732.019900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 732.037839] LNet: 45355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 732.044123] LNetError: 45355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 732.050265] LNetError: 45355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 732.057166] LNet: Removed LNI 10.1.2.3@tcp [ 732.060313] LNet: Removed LNI 192.168.206.49@tcp [ 732.075890] LNet: Removed LNI 192.168.206.49@tcp2 [ 732.099571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.179367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 732.216625] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 732.225018] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 732.231676] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 739.093542] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:05:09 (1788797109) [ 739.488932] LNet: 45952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.495649] LNetError: 45952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 739.504511] LNetError: 45952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 739.511505] LNet: Removed LNI 10.1.2.3@tcp [ 739.514353] LNet: Removed LNI 192.168.206.49@tcp [ 739.527611] LNet: Removed LNI 192.168.206.49@tcp2 [ 740.133366] Key type .llcrypt unregistered [ 740.137840] Key type ._llcrypt unregistered [ 746.409366] Key type ._llcrypt registered [ 746.412220] Key type .llcrypt registered [ 746.675197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 746.704144] alg: No test for adler32 (adler32-zlib) [ 747.663887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.749545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 747.810845] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 748.678392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.691382] LNet: 46343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.695983] LNetError: 46343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 748.711482] LNet: Removed LNI 192.168.206.49@tcp [ 748.747722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.861277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 748.881977] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 749.385439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.402878] LNet: 46373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.411827] LNetError: 46373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.429773] LNet: Removed LNI 192.168.206.49@tcp [ 749.459888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.514170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 749.533183] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 749.991365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.024210] LNet: Removed LNI 192.168.206.49@tcp [ 750.046087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.128470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 750.161333] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 750.717834] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 750.956793] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 750.975352] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 750.981119] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 753.298837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.321890] LNet: 46482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.326644] LNet: 46482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 753.331396] LNetError: 46482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.339309] LNetError: 46482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 753.353219] LNet: Removed LNI 192.168.206.49@tcp [ 753.394099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.501572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 753.535851] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 760.106490] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:05:30 (1788797130) [ 760.488878] LNet: 47079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.498690] LNetError: 47079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 760.512857] LNet: Removed LNI 192.168.206.49@tcp [ 761.077071] Key type .llcrypt unregistered [ 761.081776] Key type ._llcrypt unregistered [ 766.936455] Key type ._llcrypt registered [ 766.940252] Key type .llcrypt registered [ 767.112490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 767.125991] alg: No test for adler32 (adler32-zlib) [ 768.039570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.110438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 768.128335] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 768.942932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.967165] LNet: 47474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.974162] LNetError: 47474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.990481] LNet: Removed LNI 192.168.206.49@tcp [ 769.003377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.048310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 769.078107] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 774.411061] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:05:44 (1788797144) [ 774.694208] LNet: 48066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.703272] LNetError: 48066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.718831] LNet: Removed LNI 192.168.206.49@tcp [ 775.257485] Key type .llcrypt unregistered [ 775.259471] Key type ._llcrypt unregistered [ 780.354892] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 780.493880] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 780.499107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 780.502628] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 782.905683] Key type ._llcrypt registered [ 782.909852] Key type .llcrypt registered [ 783.106290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 783.133752] alg: No test for adler32 (adler32-zlib) [ 784.044746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.096672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 784.115959] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 784.846838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.866556] LNet: 48509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.872891] LNetError: 48509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.891526] LNet: Removed LNI 192.168.206.49@tcp [ 784.916668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.977945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 785.002111] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 790.464715] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:06:00 (1788797160) [ 790.656684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 790.670809] LNet: 49106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.674423] LNetError: 49106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.688251] LNet: Removed LNI 192.168.206.49@tcp [ 790.708523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.800606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 796.760978] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:06:06 (1788797166) [ 796.947102] LNet: 49685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.623774] Key type .llcrypt unregistered [ 797.626513] Key type ._llcrypt unregistered [ 803.625393] Key type ._llcrypt registered [ 803.627819] Key type .llcrypt registered [ 803.912598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 803.935465] alg: No test for adler32 (adler32-zlib) [ 804.841653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.901788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 804.936985] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 805.926925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.944275] LNet: 50083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.952778] LNetError: 50083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.961435] LNet: Removed LNI 192.168.206.49@tcp [ 805.980661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.038423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 806.063924] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 812.606334] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:06:22 (1788797182) [ 812.751374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.769817] LNet: 50674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.781559] LNetError: 50674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.808901] LNet: Removed LNI 192.168.206.49@tcp [ 812.829117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.898686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 818.507676] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:06:28 (1788797188) [ 820.069308] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 821.443368] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:06:31 (1788797191) [ 822.783099] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 824.281480] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:06:34 (1788797194) [ 825.592971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 827.096312] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:06:37 (1788797197) [ 828.504856] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 829.983222] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:06:39 (1788797199) [ 831.196547] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 832.659235] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:06:42 (1788797202) [ 832.829845] LNet: 52697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.401591] Key type .llcrypt unregistered [ 833.406265] Key type ._llcrypt unregistered [ 839.038694] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 839.195584] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 839.203654] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 839.229275] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 841.698860] Key type ._llcrypt registered [ 841.700895] Key type .llcrypt registered [ 841.833496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 841.847518] alg: No test for adler32 (adler32-zlib) [ 842.803794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.910973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 842.929911] LNet: Added LNI 192.168.206.49@tcp [8/512/0/180] [ 842.934913] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 842.938919] LNet: 53016:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 843.019662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.041670] LNet: 53035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.046703] LNetError: 53035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 843.058938] LNet: Removed LNI 10.1.2.3@tcp [ 843.065226] LNetError: 53035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.080813] LNet: Removed LNI 192.168.206.49@tcp [ 843.112102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.225899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 843.272940] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 843.350264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.364366] LNet: 53059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.369559] LNetError: 53059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.389255] LNet: Removed LNI 192.168.206.49@tcp [ 843.421911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.509740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 843.557176] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 843.563457] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 843.571871] LNet: 53067:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 843.682528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.720566] LNet: 53085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.728871] LNetError: 53085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 843.742273] LNet: Removed LNI 10.1.2.3@tcp [ 843.751070] LNetError: 53085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.780250] LNet: Removed LNI 192.168.206.49@tcp [ 843.831795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.925456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 843.953991] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 843.968339] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 843.970453] LNet: 53094:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 844.113662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.132939] LNet: 53120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.137835] LNetError: 53120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 844.148985] LNet: Removed LNI 10.1.2.3@tcp [ 844.151897] LNetError: 53120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.174312] LNet: Removed LNI 192.168.206.49@tcp [ 844.191360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.249628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 844.266317] LNet: Added LNI 192.168.206.49@tcp [8/256/0/182] [ 844.271842] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 844.379871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.398657] LNet: 53148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.407233] LNetError: 53148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 844.411761] LNet: Removed LNI 10.1.2.3@tcp [ 844.414322] LNetError: 53148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.424756] LNet: Removed LNI 192.168.206.49@tcp [ 844.465807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.546679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 844.575747] LNet: Added LNI 192.168.206.49@tcp [8/256/0/182] [ 844.579628] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 844.581889] LNet: 53156:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 844.662577] LNet: Removed LNI 10.1.2.3@tcp [ 844.697094] LNet: Removed LNI 192.168.206.49@tcp [ 845.301995] Key type .llcrypt unregistered [ 845.307322] Key type ._llcrypt unregistered [ 857.174755] Key type ._llcrypt registered [ 857.180595] Key type .llcrypt registered [ 858.576183] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 12:07:08 (1788797228) [ 860.018566] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 861.806788] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:07:11 (1788797231) [ 862.258797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 862.277700] alg: No test for adler32 (adler32-zlib) [ 863.343934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.398312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 863.429562] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 864.229034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.49 [ 864.246201] LNet: 54418:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 864.256109] LNet: Removed LNI 192.168.206.49@tcp [ 870.083625] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:07:19 (1788797239) [ 870.218630] LNet: 54980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.230470] LNetError: 54980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.256363] LNet: Removed LNI 192.168.206.49@tcp [ 870.828421] Key type .llcrypt unregistered [ 870.830800] Key type ._llcrypt unregistered [ 875.920678] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 876.222954] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 878.909204] Key type ._llcrypt registered [ 878.911800] Key type .llcrypt registered [ 879.153379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 879.166561] alg: No test for adler32 (adler32-zlib) [ 880.595522] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 889.147336] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:07:38 (1788797258) [ 889.233453] LNet: 55890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.240924] LNetError: 55890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 889.279673] LNet: Removed LNI 10.1.2.3@tcp [ 890.001044] Key type .llcrypt unregistered [ 890.004813] Key type ._llcrypt unregistered [ 896.245474] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 896.605212] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 896.608163] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 896.613477] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 899.314505] Key type ._llcrypt registered [ 899.316944] Key type .llcrypt registered [ 899.502859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 899.540653] alg: No test for adler32 (adler32-zlib) [ 900.528182] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 907.250901] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:07:56 (1788797276) [ 909.097269] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 910.753728] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:08:00 (1788797280) [ 910.879205] LNet: 57081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.893801] LNetError: 57081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 910.909734] LNet: Removed LNI 10.1.2.3@tcp [ 911.449569] Key type .llcrypt unregistered [ 911.452063] Key type ._llcrypt unregistered [ 917.224178] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 917.505830] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 917.510304] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 917.517057] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 920.325874] Key type ._llcrypt registered [ 920.327408] Key type .llcrypt registered [ 920.564171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 920.585151] alg: No test for adler32 (adler32-zlib) [ 921.461227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.529936] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 928.719136] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:08:18 (1788797298) [ 929.268850] LNet: 58308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.281441] LNetError: 58308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 929.301485] LNet: Removed LNI 10.1.2.3@tcp [ 929.812683] Key type .llcrypt unregistered [ 929.817696] Key type ._llcrypt unregistered [ 935.005452] Key type ._llcrypt registered [ 935.008155] Key type .llcrypt registered [ 935.271133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 935.295455] alg: No test for adler32 (adler32-zlib) [ 936.205870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 936.240391] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 943.612966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.149@tcp [ 944.309539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_interrupt -r 1 [ 944.349271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_interrupt -r 1 [ 944.393079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 944.421734] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 944.479590] LNet: 1 local NIs in recovery (showing 1): 192.168.206.49@tcp [ 944.582126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 944.613722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 944.646281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 944.670621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 944.700986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 944.739134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 944.765910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 944.796361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 944.832298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 944.854816] LNet: 58745:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 944.877169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 944.905269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 944.932821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 944.956156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 944.979964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 945.011333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 945.038229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 945.517324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_dropped -r 1 [ 945.554095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_dropped -r 1 [ 945.591258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 945.613825] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 945.809804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 945.839829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 945.865539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 945.887944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 945.914220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 945.942482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 945.976049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 946.009769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 946.034582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 946.048458] LNet: 58856:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 946.062799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 946.083759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 946.108874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 946.146640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 946.195377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 946.234897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 946.273112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 946.755613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_aborted -r 1 [ 946.781717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_aborted -r 1 [ 946.810924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 946.830420] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 947.036515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 947.080273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 947.106777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 947.129756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 947.158202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 947.192396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 947.225348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 947.259830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 947.291613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 947.304854] LNet: 58968:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 947.322415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 947.355300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 947.400210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 947.428744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 947.459421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 947.510988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 947.546913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 948.036160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_no_route -r 1 [ 948.071296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_no_route -r 1 [ 948.102381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 948.273153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 948.297827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 948.326364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 948.372672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 948.413552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 948.452430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 948.491931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 948.527383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 948.551976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 948.597438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 948.631988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 948.666509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 948.689280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 948.715505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 948.739773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 948.763923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 949.211367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_timeout -r 1 [ 949.265346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_timeout -r 1 [ 949.315438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 949.335373] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 949.341377] LNet: Skipped 1 previous similar message [ 949.554132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 949.579802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 949.606169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 949.637428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 949.660115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 949.688401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 949.714444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 949.740478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 949.764996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 949.779838] LNet: 59190:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 949.785612] LNet: 59190:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 949.799888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 949.825258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 949.848890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 949.874685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 949.904712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 949.936500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 949.960813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 950.466363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_error -r 1 [ 950.498609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_error -r 1 [ 950.528646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 950.720768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 950.752893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 950.787993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 950.825798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 950.856830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 950.884048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 950.936370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 950.981328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 951.014275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 951.055765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 951.090316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 951.132868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 951.187885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 951.232154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 951.263694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 951.297601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 951.577372] LNet: 59342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.585447] LNetError: 59342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.608078] LNet: Removed LNI 192.168.206.49@tcp [ 951.922831] Key type .llcrypt unregistered [ 951.927644] Key type ._llcrypt unregistered [ 964.905874] Key type ._llcrypt registered [ 964.908841] Key type .llcrypt registered [ 965.944389] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:08:55 (1788797335) [ 966.716292] Key type .llcrypt unregistered [ 966.721296] Key type ._llcrypt unregistered [ 971.289413] Key type ._llcrypt registered [ 971.292148] Key type .llcrypt registered [ 971.470921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 971.488213] alg: No test for adler32 (adler32-zlib) [ 972.363491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 972.392413] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 979.323657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.149@tcp [ 987.564410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.623278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 987.665270] LNet: Added LNI 192.168.206.49@tcp1 [8/256/0/180] [ 988.974891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_interrupt -r 1 [ 989.023425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e local_interrupt -r 1 [ 989.065654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_interrupt -r 1 [ 989.094810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e local_interrupt -r 1 [ 989.118898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e local_interrupt -r 1 [ 989.149784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e local_interrupt -r 1 [ 989.210440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e local_interrupt -r 1 [ 989.243921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e local_interrupt -r 1 [ 989.283602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 989.304411] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 989.311688] LNet: 1 local NIs in recovery (showing 1): 192.168.206.49@tcp [ 989.545603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 989.573859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 989.596030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 989.622480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 989.648462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 989.670379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 989.704790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 989.737293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 989.762640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 989.785806] LNet: 60807:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 989.804698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 989.850512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 989.899598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 989.928331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 989.960609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 990.004905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 990.046257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 990.521546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_dropped -r 1 [ 990.549539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e local_dropped -r 1 [ 990.617211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_dropped -r 1 [ 990.660508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e local_dropped -r 1 [ 990.693896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e local_dropped -r 1 [ 990.736958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e local_dropped -r 1 [ 990.767768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e local_dropped -r 1 [ 990.818741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e local_dropped -r 1 [ 990.846405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 990.868950] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 990.879921] LNet: Skipped 2 previous similar messages [ 991.053868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 991.078716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 991.113343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 991.152580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 991.176954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 991.207056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 991.232649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 991.269484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 991.301380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 991.322905] LNet: 60930:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 991.344902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 991.384128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 991.427987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 991.475913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 991.506263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 991.532290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 991.562666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 992.017523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_aborted -r 1 [ 992.049752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e local_aborted -r 1 [ 992.080903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_aborted -r 1 [ 992.102802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e local_aborted -r 1 [ 992.142045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e local_aborted -r 1 [ 992.184905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e local_aborted -r 1 [ 992.213712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e local_aborted -r 1 [ 992.264674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e local_aborted -r 1 [ 992.300246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 992.314865] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 992.322356] LNet: Skipped 2 previous similar messages [ 992.518904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 992.545810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 992.569287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 992.609337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 992.651136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 992.707845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 992.753805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 992.817868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 992.874615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 992.890647] LNet: 61053:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 992.904172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 992.928717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 992.955728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 992.981894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 993.012434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 993.040652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 993.074976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 993.621978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_no_route -r 1 [ 993.653831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e local_no_route -r 1 [ 993.692790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_no_route -r 1 [ 993.738661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e local_no_route -r 1 [ 993.792969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e local_no_route -r 1 [ 993.832532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e local_no_route -r 1 [ 993.865289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e local_no_route -r 1 [ 993.900313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e local_no_route -r 1 [ 993.986513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 994.254872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 994.293775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 994.319859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 994.343491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 994.364560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 994.385529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 994.421179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 994.448415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 994.472740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 994.502576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 994.527391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 994.554708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 994.596615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 994.641284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 994.680418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 994.719587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 995.218508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_timeout -r 1 [ 995.262574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e local_timeout -r 1 [ 995.290851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_timeout -r 1 [ 995.341763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e local_timeout -r 1 [ 995.397048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e local_timeout -r 1 [ 995.449850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e local_timeout -r 1 [ 995.480758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e local_timeout -r 1 [ 995.534345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e local_timeout -r 1 [ 995.562191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 995.579259] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 995.583938] LNet: Skipped 5 previous similar messages [ 995.789574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 995.822389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 995.848523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 995.878734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 995.902295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 995.933550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 995.959766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 996.003544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 996.030929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 996.050745] LNet: 61300:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 996.056354] LNet: 61300:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 996.073895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 996.109065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 996.136590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 996.162186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 996.200339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 996.234122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 996.266509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 996.755240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_error -r 1 [ 996.789585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e local_error -r 1 [ 996.831579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_error -r 1 [ 996.868405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e local_error -r 1 [ 996.904130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e local_error -r 1 [ 996.944926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e local_error -r 1 [ 996.980341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e local_error -r 1 [ 997.026227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e local_error -r 1 [ 997.073154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 997.290403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 997.316695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 997.338730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 997.360612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 997.386585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 997.407835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 997.432095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 997.455289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 997.480648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 997.512143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 997.535389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 997.565783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 997.595630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 997.630430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 997.662506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 997.712690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1002.178441] LNet: 61499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.190726] LNetError: 61499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.198991] LNet: Removed LNI 192.168.206.49@tcp [ 1002.219489] LNet: Removed LNI 192.168.206.49@tcp1 [ 1002.607964] Key type .llcrypt unregistered [ 1002.612969] Key type ._llcrypt unregistered [ 1016.716598] Key type ._llcrypt registered [ 1016.718798] Key type .llcrypt registered [ 1017.868695] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:09:47 (1788797387) [ 1018.478295] Key type .llcrypt unregistered [ 1018.482916] Key type ._llcrypt unregistered [ 1023.757970] Key type ._llcrypt registered [ 1023.760234] Key type .llcrypt registered [ 1023.918754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1023.933574] alg: No test for adler32 (adler32-zlib) [ 1024.815648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1024.835985] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1030.815366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.149@tcp [ 1031.572434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e remote_dropped -r 1 [ 1031.607214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e remote_dropped -r 1 [ 1031.647664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 1031.665535] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1031.893180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1031.923580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1031.961811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1032.001727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1032.045736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1032.118651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1032.155579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1032.217392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.247885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1032.266053] LNet: 62748:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1032.286942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1032.327752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1032.362289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1032.386768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1032.417882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1032.453579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1032.484721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1033.017687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e remote_error -r 1 [ 1033.070144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e remote_error -r 1 [ 1033.101909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 1033.120530] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1033.336238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1033.365833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1033.394558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1033.426614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1033.458350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1033.489409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1033.519795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1033.560139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1033.597924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1033.614307] LNet: 62859:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1033.629920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1033.664833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1033.718463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1033.752564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1033.799537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1033.823554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1033.849198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1034.386422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e remote_timeout -r 1 [ 1034.429878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e remote_timeout -r 1 [ 1034.459496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 1034.477246] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1034.646783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1034.674441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1034.704608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1034.730552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1034.765693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1034.788196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1034.811859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1034.837497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.871610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1034.884282] LNet: 62971:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1034.902827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.929733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.958396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1034.985775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1035.010699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1035.038205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1035.067908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1035.259910] LNet: 63012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.265842] LNetError: 63012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.275209] LNet: Removed LNI 192.168.206.49@tcp [ 1035.566423] Key type .llcrypt unregistered [ 1035.568798] Key type ._llcrypt unregistered [ 1047.544367] Key type ._llcrypt registered [ 1047.546684] Key type .llcrypt registered [ 1048.643212] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:10:18 (1788797418) [ 1049.477507] Key type .llcrypt unregistered [ 1049.481538] Key type ._llcrypt unregistered [ 1054.500555] Key type ._llcrypt registered [ 1054.502219] Key type .llcrypt registered [ 1054.622406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1054.638593] alg: No test for adler32 (adler32-zlib) [ 1055.516777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1055.538454] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1062.208894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.149@tcp [ 1071.298670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1071.357486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1071.392525] LNet: Added LNI 192.168.206.49@tcp1 [8/256/0/180] [ 1073.707677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e remote_dropped -r 1 [ 1073.772960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e remote_dropped -r 1 [ 1073.832328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e remote_dropped -r 1 [ 1073.903133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e remote_dropped -r 1 [ 1073.952262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e remote_dropped -r 1 [ 1074.006518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e remote_dropped -r 1 [ 1074.079229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e remote_dropped -r 1 [ 1074.147508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e remote_dropped -r 1 [ 1074.206183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 1074.242858] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1074.254542] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.149@tcp [ 1074.548762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1074.588783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1074.619300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1074.646635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1074.695815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1074.752500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1074.782593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1074.840774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1074.876190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1074.908723] LNet: 64474:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1074.942228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1074.985250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1075.047829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1075.108937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1075.133484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1075.177310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1075.243863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1075.294451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1075.838765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e remote_error -r 1 [ 1075.887519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e remote_error -r 1 [ 1075.920604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e remote_error -r 1 [ 1075.973300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e remote_error -r 1 [ 1076.060193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e remote_error -r 1 [ 1076.121669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e remote_error -r 1 [ 1076.169656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e remote_error -r 1 [ 1076.205656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e remote_error -r 1 [ 1076.233542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 1076.250700] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1076.260139] LNet: Skipped 2 previous similar messages [ 1076.518456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1076.568648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1076.599653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1076.620584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1076.659226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1076.693367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1076.758674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1076.823865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1076.869298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1076.906690] LNet: 64600:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1076.929745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1076.985245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1077.028655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1077.075989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1077.114468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1077.151398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1077.184754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1077.222176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1077.789800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e remote_timeout -r 1 [ 1077.821227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e remote_timeout -r 1 [ 1077.851197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e remote_timeout -r 1 [ 1077.879542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e remote_timeout -r 1 [ 1077.905960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e remote_timeout -r 1 [ 1077.931588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e remote_timeout -r 1 [ 1077.967970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e remote_timeout -r 1 [ 1078.028418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e remote_timeout -r 1 [ 1078.073115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.149@tcp [ 1078.101744] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1078.357427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1078.408437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1078.456793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1078.492203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1078.533317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1078.565482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1078.611978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1078.665963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1078.741926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1078.773069] LNet: 64725:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1078.801595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1078.831256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1078.882879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1078.920178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1078.961778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1079.001202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1079.059666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1079.111102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1084.183444] LNet: 64803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.193156] LNetError: 64803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.199203] LNet: Removed LNI 192.168.206.49@tcp [ 1084.217291] LNet: Removed LNI 192.168.206.49@tcp1 [ 1084.650695] Key type .llcrypt unregistered [ 1084.653208] Key type ._llcrypt unregistered [ 1100.516630] Key type ._llcrypt registered [ 1100.522311] Key type .llcrypt registered [ 1101.723453] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:11:11 (1788797471) [ 1103.174463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1104.817342] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:11:14 (1788797474) [ 1105.619368] Key type .llcrypt unregistered [ 1105.620907] Key type ._llcrypt unregistered [ 1110.696173] Key type ._llcrypt registered [ 1110.699748] Key type .llcrypt registered [ 1110.839249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1110.862721] alg: No test for adler32 (adler32-zlib) [ 1111.765597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1111.798668] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1117.709123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.149@tcp [ 1118.338590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e network_timeout -r 1 [ 1118.368244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e network_timeout -r 1 [ 1118.401491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.149@tcp [ 1118.431988] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1118.663918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1118.703400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1118.743163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1118.771317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1118.804762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1118.840685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1118.889871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.916888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.946702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.966880] LNet: 66342:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1118.985436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1119.009108] LNet: 1 local NIs in recovery (showing 1): 192.168.206.49@tcp [ 1119.018234] LNet: There was an unexpected network error while writing to 192.168.206.49: rc = -22 [ 1119.023714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1119.053812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1119.080452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1119.108743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.138788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1119.399345] LNet: 66381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.418745] LNetError: 66381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1119.436135] LNet: Removed LNI 192.168.206.49@tcp [ 1119.854796] Key type .llcrypt unregistered [ 1119.859049] Key type ._llcrypt unregistered [ 1134.846204] Key type ._llcrypt registered [ 1134.848315] Key type .llcrypt registered [ 1135.000187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1135.017844] alg: No test for adler32 (adler32-zlib) [ 1135.941143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1135.968328] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1142.110627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.149@tcp [ 1149.124787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1149.169252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1149.188196] LNet: Added LNI 192.168.206.49@tcp1 [8/256/0/180] [ 1150.274304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e network_timeout -r 1 [ 1150.323084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e network_timeout -r 1 [ 1150.355119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e network_timeout -r 1 [ 1150.383569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e network_timeout -r 1 [ 1150.416964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e network_timeout -r 1 [ 1150.447423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e network_timeout -r 1 [ 1150.484648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e network_timeout -r 1 [ 1150.520262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e network_timeout -r 1 [ 1150.558863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.149@tcp [ 1150.580952] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1150.727383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1150.757162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1150.782275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1150.811275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1150.857382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1150.905890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1150.935224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1150.979522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1151.023973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1151.040065] LNet: 67255:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1151.055095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1151.080358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1151.107735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1151.133331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1151.157299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1151.180256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1154.778622] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1154.986501] LNet: 67329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.992244] LNetError: 67329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1154.999076] LNet: Removed LNI 192.168.206.49@tcp [ 1155.013519] LNet: Removed LNI 192.168.206.49@tcp1 [ 1155.330698] Key type .llcrypt unregistered [ 1155.337639] Key type ._llcrypt unregistered [ 1168.948484] Key type ._llcrypt registered [ 1168.950193] Key type .llcrypt registered [ 1170.162142] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:12:19 (1788797539) [ 1170.477551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1170.502529] alg: No test for adler32 (adler32-zlib) [ 1171.381805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1171.428135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1171.451737] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1172.026914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1172.051170] LNet: Added LNI 192.168.206.49@tcp1 [8/256/0/180] [ 1172.720315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1172.775651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1172.820724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1172.851128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1172.923480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1172.983268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1173.020533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.49@tcp [ 1173.054658] LNetError: 68425:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.49@tcp to recovery [ 1173.472135] LNet: 1 local NIs in recovery (showing 1): 192.168.206.49@tcp [ 1173.472398] LNet: There was an unexpected network error while writing to 192.168.206.49: rc = -22 [ 1175.520622] LNet: There was an unexpected network error while writing to 192.168.206.49: rc = -22 [ 1176.457518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1179.615570] LNet: There was an unexpected network error while writing to 192.168.206.49: rc = -22 [ 1179.877507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1179.907225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1179.952501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1179.967373] LNet: 68532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.972449] LNetError: 68532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1179.977412] LNet: Removed LNI 192.168.206.49@tcp [ 1179.996188] LNet: Removed LNI 192.168.206.49@tcp1 [ 1180.012619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1180.051705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1180.071656] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1180.635171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1180.662669] LNet: Added LNI 192.168.206.49@tcp1 [8/256/0/180] [ 1181.314385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1181.345806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.389357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1181.443957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1181.473244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1181.511214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1181.547577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1181.575823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.49@tcp [ 1181.589636] LNetError: 68804:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.49@tcp to recovery [ 1182.111476] LNet: There was an unexpected network error while writing to 192.168.206.49: rc = -22 [ 1185.002347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1188.255376] LNet: There was an unexpected network error while writing to 192.168.206.49: rc = -22 [ 1188.262820] LNet: Skipped 1 previous similar message [ 1192.878371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1192.916640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1192.949465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1198.221742] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:12:48 (1788797568) [ 1198.371347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1198.388061] LNet: 69521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.397723] LNetError: 69521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1198.403242] LNetError: 69521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1198.410183] LNet: Removed LNI 192.168.206.49@tcp [ 1198.420809] LNet: Removed LNI 192.168.206.49@tcp1 [ 1198.443563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1198.490485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1198.530872] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1199.081939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1199.116434] LNet: Added LNI 192.168.206.49@tcp1 [8/256/0/180] [ 1199.804805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1199.861658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1199.898639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1199.955064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1199.989874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1200.031815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1200.062661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.49@tcp --health 0 [ 1200.164381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1200.543155] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.49@tcp [ 1201.567491] LNet: There was an unexpected network error while writing to 192.168.206.49: rc = -22 [ 1201.579910] LNet: Skipped 1 previous similar message [ 1216.582230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1216.737694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1216.766909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1216.788605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.49@tcp --health 500 [ 1216.879431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1222.424708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1222.487109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1222.507531] LNet: 70012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1222.512331] LNetError: 70012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1222.516488] LNetError: 70012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1222.520700] LNet: Removed LNI 192.168.206.49@tcp [ 1222.536296] LNet: Removed LNI 192.168.206.49@tcp1 [ 1222.554788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1222.587550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1222.610919] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1223.521357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1223.545270] LNet: Added LNI 192.168.206.49@tcp1 [8/256/0/180] [ 1224.572106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1224.606432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1224.668813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1224.700428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1224.732815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1224.770979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1224.803794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1224.835282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.49@tcp --health 0 [ 1225.631384] LNet: There was an unexpected network error while writing to 192.168.206.49: rc = -22 [ 1225.637218] LNet: Skipped 3 previous similar messages [ 1226.247170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1236.549209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1236.597325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1236.664294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1242.903669] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:13:32 (1788797612) [ 1243.278682] LNet: 71058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.286366] LNetError: 71058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1243.292905] LNetError: 71058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1243.303132] LNet: Removed LNI 192.168.206.49@tcp [ 1243.319395] LNet: Removed LNI 192.168.206.49@tcp1 [ 1243.837539] Key type .llcrypt unregistered [ 1243.844345] Key type ._llcrypt unregistered [ 1248.870538] Key type ._llcrypt registered [ 1248.875678] Key type .llcrypt registered [ 1249.016722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1249.034684] alg: No test for adler32 (adler32-zlib) [ 1249.917337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1249.937679] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1257.816932] Lustre: DEBUG MARKER: Initial discovery [ 1257.918052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.149@tcp [ 1259.844577] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1259.909430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1259.951803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.149@tcp [ 1259.994816] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1260.012279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1260.042278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.149@tcp -r 1 -m GET -l 3 [ 1260.081851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.149@tcp -r 1 -m GET -e local_error [ 1260.117379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.149@tcp -r 1 -m PUT -l 6 [ 1260.192576] LNet: 1 local NIs in recovery (showing 1): 192.168.206.49@tcp [ 1261.414279] Lustre: DEBUG MARKER: Force oleg649-client.virtnet to discover 192.168.206.149@tcp (in background) [ 1261.461950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.149@tcp [ 1262.850881] Lustre: DEBUG MARKER: Force oleg649-server to discover 192.168.206.49@tcp [ 1264.545096] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1265.232514] Lustre: DEBUG MARKER: Wait for 71808 [ 1266.509596] Lustre: DEBUG MARKER: Finished wait on 71808 [ 1266.730354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1266.777596] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1266.786680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1267.095641] LNet: 72266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1267.100926] LNetError: 72266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1267.119044] LNet: Removed LNI 192.168.206.49@tcp [ 1267.500507] Key type .llcrypt unregistered [ 1267.502665] Key type ._llcrypt unregistered [ 1280.746203] Key type ._llcrypt registered [ 1280.748556] Key type .llcrypt registered [ 1281.654322] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:14:11 (1788797651) [ 1282.072527] Key type .llcrypt unregistered [ 1282.078320] Key type ._llcrypt unregistered [ 1286.037749] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1286.192808] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1286.195801] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1286.198535] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1288.502714] Key type ._llcrypt registered [ 1288.504762] Key type .llcrypt registered [ 1288.623957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1288.635600] alg: No test for adler32 (adler32-zlib) [ 1289.510319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1289.555178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1289.572394] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1290.227565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1290.246454] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1297.165551] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:14:26 (1788797666) [ 1297.275917] LNet: 74327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.283199] LNetError: 74327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1297.287552] LNet: Removed LNI 10.1.2.3@tcp [ 1297.294950] LNet: Removed LNI 192.168.206.49@tcp [ 1297.921392] Key type .llcrypt unregistered [ 1297.923970] Key type ._llcrypt unregistered [ 1303.215685] Key type ._llcrypt registered [ 1303.219963] Key type .llcrypt registered [ 1303.366986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1303.382284] alg: No test for adler32 (adler32-zlib) [ 1304.365549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1304.479716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1304.503292] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1305.119647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1305.132825] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1305.962200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.49@tcp [ 1307.039191] LNetError: Unexpected error -22 connecting to 192.168.206.49@tcp at host 192.168.206.49:988 [ 1311.643368] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:14:41 (1788797681) [ 1311.842874] LNet: 75456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.852736] LNetError: 75456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1311.864384] LNet: Removed LNI 10.1.2.3@tcp [ 1311.879407] LNet: Removed LNI 192.168.206.49@tcp [ 1312.472470] Key type .llcrypt unregistered [ 1312.474742] Key type ._llcrypt unregistered [ 1317.718863] Key type ._llcrypt registered [ 1317.721313] Key type .llcrypt registered [ 1317.885918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1317.899858] alg: No test for adler32 (adler32-zlib) [ 1318.783701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1318.841398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1318.860871] LNet: Added LNI 192.168.206.49@tcp1 [8/256/0/180] [ 1319.377032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1319.408648] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 1320.181560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.49@tcp1 --nid 192.168.206.49@tcp2 [ 1323.629566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.49@tcp1 192.168.206.49@tcp2 [ 1323.658447] LNetError: 76290:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.49@tcp1 to 192.168.206.49@tcp2 - Route table may be misconfigured [ 1323.691686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.49@tcp2 192.168.206.49@tcp1 [ 1328.445705] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:14:58 (1788797698) [ 1328.544622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1328.577740] LNet: 76857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1328.587222] LNetError: 76857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1328.595141] LNet: Removed LNI 192.168.206.49@tcp1 [ 1329.636911] LNet: Removed LNI 192.168.206.49@tcp2 [ 1329.660439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1329.704970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1329.724063] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1330.273634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1330.288815] LNet: Added LNI 192.168.206.49@tcp1 [8/256/0/180] [ 1330.958806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1330.983987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.49@tcp 192.168.206.49@tcp [ 1331.012843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.49@tcp1 192.168.206.49@tcp1 [ 1331.039895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.49@tcp -d 192.168.206.49@tcp -e network_timeout [ 1331.067878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -e network_timeout [ 1331.097346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -e network_timeout [ 1331.134952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -e network_timeout [ 1331.166733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1331.185982] LNet: There was an unexpected network error while writing to 192.168.206.49: rc = -22 [ 1331.202624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1331.271958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1331.334569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1331.743343] LNet: 1 local NIs in recovery (showing 1): 192.168.206.49@tcp [ 1335.515673] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:15:05 (1788797705) [ 1335.608052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1335.644025] LNet: 77720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.652808] LNetError: 77720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1335.657718] LNetError: 77720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1335.663149] LNet: Removed LNI 192.168.206.49@tcp [ 1336.680580] LNet: Removed LNI 192.168.206.49@tcp1 [ 1336.722619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1336.816775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1337.067271] LNet: 77763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1337.563688] Key type .llcrypt unregistered [ 1337.570329] Key type ._llcrypt unregistered [ 1346.954253] Key type ._llcrypt registered [ 1346.956429] Key type .llcrypt registered [ 1347.880982] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:15:17 (1788797717) [ 1348.146254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1348.155971] alg: No test for adler32 (adler32-zlib) [ 1349.070771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1350.236867] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1351.958116] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:15:21 (1788797721) [ 1352.039589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1352.084041] LNet: 78850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1352.109633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1352.167834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1352.191254] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1352.733787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1352.773210] LNet: Added LNI 192.168.206.49@tcp1 [8/256/0/180] [ 1353.472707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1353.530473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp1 [ 1353.570524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp1 [ 1358.933515] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:15:28 (1788797728) [ 1360.505992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg649-server [ 1362.054916] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:15:31 (1788797731) [ 1363.522666] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg649-server [ 1365.158511] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:15:34 (1788797734) [ 1366.891761] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg649-server [ 1368.320492] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:15:38 (1788797738) [ 1369.782386] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg649-server [ 1371.441171] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:15:41 (1788797741) [ 1373.134846] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg649-server [ 1375.453651] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:15:44 (1788797744) [ 1376.903910] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg649-server [ 1378.224591] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:15:48 (1788797748) [ 1379.744897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg649-server [ 1381.124758] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:15:51 (1788797751) [ 1382.469347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg649-server [ 1384.022226] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:15:53 (1788797753) [ 1385.619355] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg649-server [ 1387.180511] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:15:57 (1788797757) [ 1387.457441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1387.480896] LNet: 82734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.486915] LNetError: 82734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.511153] LNet: Removed LNI 192.168.206.49@tcp [ 1388.516638] LNetError: 82734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.532633] LNet: Removed LNI 192.168.206.49@tcp1 [ 1388.570227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1388.629802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1388.670360] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1389.508280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1389.579552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1389.732862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1389.758409] LNet: 82882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.767588] LNetError: 82882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.782970] LNet: Removed LNI 192.168.206.49@tcp [ 1389.809425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1389.863129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1389.889547] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1390.455350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1390.596953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1391.658692] LNet: Removed LNI 192.168.206.49@tcp [ 1391.674487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1391.748668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1391.771530] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1392.314717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1392.378262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1392.599814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1392.624774] LNet: 83183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1392.633563] LNet: 83183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1392.643669] LNetError: 83183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.652488] LNetError: 83183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1392.669641] LNet: Removed LNI 192.168.206.49@tcp [ 1392.697443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1392.772960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1392.800767] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1393.489039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1393.800001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1394.869906] LNet: Removed LNI 192.168.206.49@tcp [ 1394.947961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1395.037797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1395.064391] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1395.757826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1395.837222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1397.249083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1397.273443] LNet: 83485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1397.280595] LNet: 83485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1397.288876] LNetError: 83485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1397.299249] LNetError: 83485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1397.315246] LNet: Removed LNI 192.168.206.49@tcp [ 1397.376574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1397.428116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1397.458317] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1398.098421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1398.349718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1399.464939] LNet: Removed LNI 192.168.206.49@tcp [ 1399.491724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1399.550727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1399.617905] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1400.427187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1400.518948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1400.864805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1401.958585] LNet: Removed LNI 192.168.206.49@tcp [ 1401.986718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1402.051498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1402.075960] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1402.807441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1403.074629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1404.135927] LNet: Removed LNI 192.168.206.49@tcp [ 1404.183979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1404.297324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1404.331437] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1405.147880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1405.225538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1406.665597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1406.684396] LNet: 84079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.695380] LNet: 84079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1406.706580] LNetError: 84079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.713638] LNetError: 84079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1406.731244] LNet: Removed LNI 192.168.206.49@tcp [ 1406.765084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1406.806495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1406.829287] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1407.571785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1407.874498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1408.933904] LNet: Removed LNI 192.168.206.49@tcp [ 1408.954099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1408.998295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1409.023994] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1409.798637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1409.869520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1410.135750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1411.244229] LNet: Removed LNI 192.168.206.49@tcp [ 1411.290588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.399958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1411.429748] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1412.256292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1412.532866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1413.606083] LNet: Removed LNI 192.168.206.49@tcp [ 1413.622233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1413.666297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1413.693054] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1414.343159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1414.413620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1414.433648] LNet: Removed LNI 192.168.206.49@tcp [ 1414.444876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1414.477440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1414.490568] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1415.141758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1415.210366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1415.233199] LNet: Removed LNI 192.168.206.49@tcp [ 1415.253855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1415.296482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1415.356735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1415.388510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1415.429804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1415.510414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1415.540541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1415.608373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1415.716734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1415.746187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1415.794761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1415.827952] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1416.440512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1416.463939] LNet: Removed LNI 192.168.206.49@tcp [ 1416.477885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1416.531978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1422.595798] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:16:32 (1788797792) [ 1422.689056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1422.739781] LNet: 85534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.761237] LNet: 85534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1422.799981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1422.865391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1422.889873] LNet: Added LNI 192.168.206.49@tcp231 [8/256/0/180] [ 1423.998936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1424.026539] LNetError: 85675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1424.032469] LNetError: 85675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1424.074909] LNet: Removed LNI 192.168.206.49@tcp231 [ 1424.104894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1424.189619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1424.214668] LNet: Added LNI 192.168.206.49@tcp231 [8/256/0/180] [ 1424.218405] LNet: 85683:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1424.718490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1424.748220] LNet: Removed LNI 192.168.206.49@tcp231 [ 1424.768714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1424.804467] LNet: Added LNI 192.168.206.49@tcp231 [8/256/0/180] [ 1425.909273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1425.943709] LNet: Removed LNI 192.168.206.49@tcp231 [ 1425.968914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1426.045937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1426.073146] LNet: Added LNI 192.168.206.49@tcp231 [8/256/0/0] [ 1426.078990] LNet: 85836:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1426.355224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1426.373219] LNet: Removed LNI 192.168.206.49@tcp231 [ 1426.390823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1426.421990] LNet: Added LNI 192.168.206.49@tcp231 [8/256/0/0] [ 1427.317845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1427.402703] LNet: Removed LNI 192.168.206.49@tcp231 [ 1427.449070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1427.526811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1427.560431] LNet: Added LNI 192.168.206.49@tcp231 [8/256/0/60] [ 1427.565190] LNet: 85987:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1427.943492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1427.985128] LNet: Removed LNI 192.168.206.49@tcp231 [ 1427.997434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1428.023952] LNet: Added LNI 192.168.206.49@tcp231 [8/256/0/60] [ 1428.934933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1428.976303] LNet: Removed LNI 192.168.206.49@tcp231 [ 1429.001789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1429.086361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1429.106779] LNet: Added LNI 192.168.206.49@tcp231 [8/256/0/180] [ 1429.497603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1429.526729] LNet: Removed LNI 192.168.206.49@tcp231 [ 1429.537977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1429.570553] LNet: Added LNI 192.168.206.49@tcp231 [8/256/0/180] [ 1435.234406] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:16:45 (1788797805) [ 1435.306098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1435.344691] LNet: Removed LNI 192.168.206.49@tcp231 [ 1435.379667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.460399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1435.490067] LNet: Added LNI 192.168.206.49@tcp232 [8/256/0/180] [ 1436.153198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1436.178922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1436.206241] LNet: Removed LNI 192.168.206.49@tcp232 [ 1436.215801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1436.233517] LNet: Added LNI 192.168.206.49@tcp232 [8/256/0/180] [ 1436.812321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1436.863781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1436.900191] LNet: Removed LNI 192.168.206.49@tcp232 [ 1436.951871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1437.006269] LNet: Added LNI 192.168.206.49@tcp232 [8/256/0/180] [ 1437.701792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1437.743903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1437.771537] LNet: Removed LNI 192.168.206.49@tcp232 [ 1437.785225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1437.827452] LNet: Added LNI 192.168.206.49@tcp232 [8/256/0/180] [ 1438.519370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1438.566149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1438.602634] LNet: Removed LNI 192.168.206.49@tcp232 [ 1438.627835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1444.605649] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:16:54 (1788797814) [ 1445.640974] Key type .llcrypt unregistered [ 1445.644514] Key type ._llcrypt unregistered [ 1450.594920] Key type ._llcrypt registered [ 1450.601974] Key type .llcrypt registered [ 1450.743557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1450.755850] alg: No test for adler32 (adler32-zlib) [ 1451.626212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1451.644982] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1458.403450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.149@tcp [ 1466.633165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1466.694577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1466.734546] LNet: Added LNI 192.168.206.49@tcp1 [8/256/0/180] [ 1468.031788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_interrupt -r 2 [ 1468.085768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e local_interrupt -r 2 [ 1468.128318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_interrupt -r 2 [ 1468.189626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e local_interrupt -r 2 [ 1468.245671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e local_interrupt -r 2 [ 1468.281272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e local_interrupt -r 2 [ 1468.309185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e local_interrupt -r 2 [ 1468.357185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e local_interrupt -r 2 [ 1468.402689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.49@tcp 192.168.206.149@tcp [ 1468.423928] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1468.432030] LNet: 1 local NIs in recovery (showing 1): 192.168.206.49@tcp [ 1468.657563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1468.682784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1468.706336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1468.731922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1468.756623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1468.790297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1468.817112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1468.848545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1468.878361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1468.901476] LNet: 88636:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1468.921593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1468.972466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1469.013286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1469.067191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1469.123746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1469.172762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1469.561818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_dropped -r 2 [ 1469.599687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e local_dropped -r 2 [ 1469.640323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_dropped -r 2 [ 1469.687437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e local_dropped -r 2 [ 1469.739610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e local_dropped -r 2 [ 1469.766995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e local_dropped -r 2 [ 1469.797951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e local_dropped -r 2 [ 1469.832228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e local_dropped -r 2 [ 1469.878428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.49@tcp 192.168.206.149@tcp [ 1469.921354] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1470.164403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1470.220762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1470.257940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1470.290488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1470.319850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1470.346992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1470.382456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1470.431692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1470.464679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1470.502775] LNet: 88759:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1470.529445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1470.557371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1470.588403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1470.623389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1470.657996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1470.695312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1471.119549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_aborted -r 2 [ 1471.158368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e local_aborted -r 2 [ 1471.195233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_aborted -r 2 [ 1471.238766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e local_aborted -r 2 [ 1471.274170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e local_aborted -r 2 [ 1471.302922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e local_aborted -r 2 [ 1471.343608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e local_aborted -r 2 [ 1471.402785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e local_aborted -r 2 [ 1471.463398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.49@tcp 192.168.206.149@tcp [ 1471.493443] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1471.734557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1471.761742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1471.795955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1471.862183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1471.916350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1471.953539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1471.991069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1472.035316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1472.095573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1472.121808] LNet: 88881:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1472.149924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1472.205367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1472.241395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1472.294279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1472.341345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1472.415701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1472.908210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_no_route -r 2 [ 1472.935378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e local_no_route -r 2 [ 1472.973995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_no_route -r 2 [ 1473.015928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e local_no_route -r 2 [ 1473.052724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e local_no_route -r 2 [ 1473.091756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e local_no_route -r 2 [ 1473.148110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e local_no_route -r 2 [ 1473.183178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e local_no_route -r 2 [ 1473.241190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.49@tcp 192.168.206.149@tcp [ 1473.571247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1473.602559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1473.643271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1473.688255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1473.708836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1473.732372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1473.772735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1473.806617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1473.840218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1473.877330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1473.909886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1473.947258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1473.983777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1474.044327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1474.101598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1474.550208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e local_timeout -r 2 [ 1474.597618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e local_timeout -r 2 [ 1474.655238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e local_timeout -r 2 [ 1474.702160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e local_timeout -r 2 [ 1474.743323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e local_timeout -r 2 [ 1474.775908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e local_timeout -r 2 [ 1474.826238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e local_timeout -r 2 [ 1474.856739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e local_timeout -r 2 [ 1474.903250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.49@tcp 192.168.206.149@tcp [ 1474.935480] LNet: There was an unexpected network error while writing to 192.168.206.149: rc = -22 [ 1474.939891] LNet: Skipped 1 previous similar message [ 1475.167484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1475.191207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1475.228813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1475.256911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1475.279106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1475.307219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1475.336705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1475.367780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1475.395575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1475.409502] LNet: 89125:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1475.413975] LNet: 89125:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1475.429524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1475.461911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1475.492910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1475.531436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1475.560483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1475.593308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1476.049976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp -m GET -e remote_dropped -r 2 [ 1476.079840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.149@tcp1 -m GET -e remote_dropped -r 2 [ 1476.107863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp -m GET -e remote_dropped -r 2 [ 1476.143097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp -d 192.168.206.49@tcp1 -m GET -e remote_dropped -r 2 [ 1476.173749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp -m GET -e remote_dropped -r 2 [ 1476.204729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.149@tcp1 -m GET -e remote_dropped -r 2 [ 1476.233488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp -m GET -e remote_dropped -r 2 [ 1476.267294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.49@tcp1 -d 192.168.206.49@tcp1 -m GET -e remote_dropped -r 2 [ 1476.305407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.49@tcp 192.168.206.149@tcp [ 1476.331672] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.149@tcp [ 1476.540114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1476.586400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1476.622462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1476.663898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1476.701586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1476.739445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1476.770469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1476.804916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1476.835546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1476.880403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1476.922870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1476.952479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1476.977544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1477.002166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1477.023854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1481.830523] LNet: 89321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.848356] LNetError: 89321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.857245] LNet: Removed LNI 192.168.206.49@tcp [ 1481.872934] LNet: Removed LNI 192.168.206.49@tcp1 [ 1482.213090] Key type .llcrypt unregistered [ 1482.215356] Key type ._llcrypt unregistered [ 1496.709931] Key type ._llcrypt registered [ 1496.711695] Key type .llcrypt registered [ 1497.838982] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:17:47 (1788797867) [ 1498.411588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1498.424982] alg: No test for adler32 (adler32-zlib) [ 1499.334237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.400231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1499.419297] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1500.282703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1500.342502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.49@tcp [ 1500.391891] LNetError: 90297:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.49@tcp to recovery [ 1500.525714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.49@tcp [ 1500.706980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1507.330990] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:17:56 (1788797876) [ 1507.478071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1507.495989] LNet: 90880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1507.504221] LNetError: 90880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1507.519361] LNet: Removed LNI 192.168.206.49@tcp [ 1507.543054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1507.613475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1507.633032] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1508.286340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1508.309849] LNet: Added LNI 192.168.206.49@tcp1 [8/256/0/180] [ 1509.148235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.49@tcp [ 1509.200171] LNetError: 91153:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.49@tcp to recovery [ 1509.663744] LNet: 1 local NIs in recovery (showing 1): 192.168.206.49@tcp [ 1509.666716] LNet: There was an unexpected network error while writing to 192.168.206.49: rc = -22 [ 1510.473555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.49@tcp [ 1518.393851] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:18:08 (1788797888) [ 1520.130833] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg649-server [ 1521.821279] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:18:11 (1788797891) [ 1523.415960] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg649-server [ 1525.251270] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:18:14 (1788797894) [ 1526.787180] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg649-server [ 1528.469944] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:18:18 (1788797898) [ 1528.569471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.589497] LNet: 92775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.601490] LNetError: 92775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1528.609808] LNet: Removed LNI 192.168.206.49@tcp [ 1528.619172] LNet: Removed LNI 192.168.206.49@tcp1 [ 1528.679834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.733581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1528.768929] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1529.396902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1529.446734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1529.490443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1529.577887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1529.617266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1529.697290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1535.267226] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:18:25 (1788797905) [ 1535.381058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.422465] LNet: 93507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.439480] LNetError: 93507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.454320] LNetError: 93507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1535.485116] LNet: Removed LNI 192.168.206.49@tcp [ 1535.523751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.591213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1535.623248] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1542.207976] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:18:32 (1788797912) [ 1542.524053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.566382] LNet: 94217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.573768] LNetError: 94217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1542.591407] LNet: Removed LNI 192.168.206.49@tcp [ 1542.621770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.688534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1542.725780] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1548.369581] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:18:38 (1788797918) [ 1549.851403] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1551.387581] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:18:41 (1788797921) [ 1551.967801] LNet: 95251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1551.974230] LNetError: 95251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1551.993564] LNet: Removed LNI 192.168.206.49@tcp [ 1552.629846] Key type .llcrypt unregistered [ 1552.633718] Key type ._llcrypt unregistered [ 1558.246386] Key type ._llcrypt registered [ 1558.249395] Key type .llcrypt registered [ 1558.445131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1558.460592] alg: No test for adler32 (adler32-zlib) [ 1559.444603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1559.468850] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1566.692316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.149@tcp [ 1571.912927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.149@tcp [ 1572.218754] LNet: 95666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.233015] LNetError: 95666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1572.251755] LNet: Removed LNI 192.168.206.49@tcp [ 1572.603134] Key type .llcrypt unregistered [ 1572.604877] Key type ._llcrypt unregistered [ 1582.038271] Key type ._llcrypt registered [ 1582.039810] Key type .llcrypt registered [ 1583.209779] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1584.726250] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1586.565631] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:19:16 (1788797956) [ 1587.976889] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1589.735291] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:19:19 (1788797959) [ 1591.546523] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg649-server [ 1592.945667] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:19:22 (1788797962) [ 1594.723996] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg649-server [ 1596.309398] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:19:26 (1788797966) [ 1597.875776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1597.897498] alg: No test for adler32 (adler32-zlib) [ 1598.935737] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 1599.177095] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1600.907725] Key type lgssc registered [ 1601.983939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1620.169350] Key type lgssc unregistered [ 1620.387617] LNet: 98836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.392957] LNetError: 98836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1620.413300] LNet: Removed LNI 192.168.206.49@tcp [ 1621.010522] Key type .llcrypt unregistered [ 1621.012340] Key type ._llcrypt unregistered [ 1633.613675] Key type ._llcrypt registered [ 1633.616243] Key type .llcrypt registered [ 1634.718967] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:20:04 (1788798004) [ 1635.100977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1635.131670] alg: No test for adler32 (adler32-zlib) [ 1636.051533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.151595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1636.191842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1636.229860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1636.263388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1641.602463] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:20:11 (1788798011) [ 1641.723929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.744121] LNet: 100248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.788729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.854160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1641.882148] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1642.730283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.760410] LNet: 100394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1642.767716] LNetError: 100394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1642.795236] LNet: Removed LNI 192.168.206.49@tcp [ 1642.823717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.869985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1642.925309] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1643.507460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1643.526766] LNetError: 100422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1643.543281] LNet: Removed LNI 192.168.206.49@tcp [ 1643.566227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.624647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1643.644469] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1649.607158] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:20:19 (1788798019) [ 1649.748074] LNet: 101009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1649.752670] LNet: 101009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1649.765129] LNetError: 101009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1649.792796] LNet: Removed LNI 192.168.206.49@tcp [ 1650.368534] Key type .llcrypt unregistered [ 1650.380034] Key type ._llcrypt unregistered [ 1655.675994] Key type ._llcrypt registered [ 1655.677582] Key type .llcrypt registered [ 1655.854060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1655.869361] alg: No test for adler32 (adler32-zlib) [ 1656.773487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.880388] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1656.884108] LNet: *** cfs_fail_loc=e003, val=100*** [ 1657.569890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1657.584523] LNet: *** cfs_fail_loc=e003, val=100*** [ 1657.587361] LNet: Skipped 2 previous similar messages [ 1657.594301] Ping reply received with mapping: [ 1657.598714] found metadata reply NI[192.168.206.49@tcp] [ 1657.610029] 31 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1657.612828] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1657.617480] 5a 5a 5a 5a 5a 5a 5a 5a [ 1657.776289] LNet: 101410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.782289] LNetError: 101410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1657.794546] LNet: Removed LNI 192.168.206.49@tcp [ 1658.309091] Key type .llcrypt unregistered [ 1658.311530] Key type ._llcrypt unregistered [ 1668.386891] Key type ._llcrypt registered [ 1668.390457] Key type .llcrypt registered [ 1669.794907] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:20:39 (1788798039) [ 1670.169305] Key type .llcrypt unregistered [ 1670.173249] Key type ._llcrypt unregistered [ 1676.335441] Key type ._llcrypt registered [ 1676.337231] Key type .llcrypt registered [ 1676.500074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1676.524611] alg: No test for adler32 (adler32-zlib) [ 1677.545763] LNetError: 102447:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1677.865516] Key type .llcrypt unregistered [ 1677.869871] Key type ._llcrypt unregistered [ 1682.524241] Key type ._llcrypt registered [ 1682.527565] Key type .llcrypt registered [ 1683.775299] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:20:53 (1788798053) [ 1684.386425] Key type .llcrypt unregistered [ 1684.389301] Key type ._llcrypt unregistered [ 1690.656380] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1690.665535] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1693.324299] Key type ._llcrypt registered [ 1693.327916] Key type .llcrypt registered [ 1693.465498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1693.476680] alg: No test for adler32 (adler32-zlib) [ 1694.470186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.543632] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1705.615534] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:21:15 (1788798075) [ 1705.702668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.762740] LNet: 104214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.773995] LNetError: 104214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1705.790564] LNet: Removed LNI 10.1.2.3@tcp [ 1705.837612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.861241] Key type .llcrypt unregistered [ 1706.864162] Key type ._llcrypt unregistered [ 1713.648723] Key type ._llcrypt registered [ 1713.652131] Key type .llcrypt registered [ 1714.012374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1714.085620] alg: No test for adler32 (adler32-zlib) [ 1715.107617] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1715.231585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1715.426093] LNet: 104505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.446956] LNetError: 104505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.464240] LNet: Removed LNI 192.168.206.49@tcp [ 1716.405416] Key type .llcrypt unregistered [ 1716.407192] Key type ._llcrypt unregistered [ 1729.020943] Key type ._llcrypt registered [ 1729.024052] Key type .llcrypt registered [ 1730.802841] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:21:39 (1788798099) [ 1731.752484] Key type .llcrypt unregistered [ 1731.764774] Key type ._llcrypt unregistered [ 1737.429444] Key type ._llcrypt registered [ 1737.432087] Key type .llcrypt registered [ 1737.630441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1737.644521] alg: No test for adler32 (adler32-zlib) [ 1744.834306] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:21:54 (1788798114) [ 1744.923908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.988992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.085259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1745.104720] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1745.975244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1746.003394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1746.040679] LNet: Removed LNI 192.168.206.49@tcp [ 1746.280740] LNet: 106246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.637080] Key type .llcrypt unregistered [ 1746.638730] Key type ._llcrypt unregistered [ 1757.395325] Key type ._llcrypt registered [ 1757.399949] Key type .llcrypt registered [ 1758.569741] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:22:08 (1788798128) [ 1759.080856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1759.101318] alg: No test for adler32 (adler32-zlib) [ 1760.052353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.105898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1760.125925] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1760.907648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1760.947727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.49@tcp [ 1760.968425] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1760.976684] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1760.979984] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1760, 0, 0 [ 1760.993432] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1760.998931] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1761.004876] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1761.007761] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1760, 0, 0 [ 1761.022954] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1761.027864] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1761.032284] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1761.035362] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1760, 0, 1810 [ 1761.038936] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1761.040824] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1761.043435] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1761.046531] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1760, 0, 1810 [ 1761.051331] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1768.003166] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:22:17 (1788798137) [ 1768.610889] LNet: 107783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1768.618697] LNetError: 107783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.642699] LNet: Removed LNI 192.168.206.49@tcp [ 1770.313746] Key type .llcrypt unregistered [ 1770.319068] Key type ._llcrypt unregistered [ 1776.296881] Key type ._llcrypt registered [ 1776.303333] Key type .llcrypt registered [ 1776.601269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1776.637561] alg: No test for adler32 (adler32-zlib) [ 1777.663190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1777.687568] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1785.527629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.149@tcp [ 1794.437380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.496269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1794.518506] LNet: Added LNI 192.168.206.49@tcp1 [8/256/0/180] [ 1795.586927] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1795.768297] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1795.782629] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1795.787099] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1798.160714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1798.183466] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1808.560309] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:22:58 (1788798178) [ 1808.875818] LNet: 109283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.885159] LNetError: 109283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.893328] LNet: Removed LNI 192.168.206.49@tcp [ 1809.952270] LNet: Removed LNI 192.168.206.49@tcp1 [ 1809.961143] LNetError: 109283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1809.971728] LNetError: 109283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1810.004261] LNet: Removed LNI 10.1.2.3@tcp99 [ 1810.858704] Key type .llcrypt unregistered [ 1810.861375] Key type ._llcrypt unregistered [ 1817.359716] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1817.364488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1820.035623] Key type ._llcrypt registered [ 1820.043401] Key type .llcrypt registered [ 1820.377122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1820.412691] alg: No test for adler32 (adler32-zlib) [ 1821.435443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.538904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.582308] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1822.469204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1822.495346] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1824.771423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1824.909825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1825.047223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1825.100585] LNetError: 110024:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1825.174421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1825.243330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1825.681048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1825.814478] LNet: 110044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.819721] LNetError: 110044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1825.825498] LNet: Removed LNI 10.1.2.3@tcp [ 1826.858834] LNet: Removed LNI 192.168.206.49@tcp [ 1827.591488] Key type .llcrypt unregistered [ 1827.593270] Key type ._llcrypt unregistered [ 1839.127934] Key type ._llcrypt registered [ 1839.129918] Key type .llcrypt registered [ 1840.598655] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:23:30 (1788798210) [ 1840.963250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1840.986752] alg: No test for adler32 (adler32-zlib) [ 1841.893517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.963479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1841.980971] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1842.773523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1842.871600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg649-client@tcp [ 1849.223450] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:23:39 (1788798219) [ 1849.611483] LNet: 111574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.623375] LNetError: 111574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1849.654033] LNet: Removed LNI 192.168.206.49@tcp [ 1850.394754] Key type .llcrypt unregistered [ 1850.404505] Key type ._llcrypt unregistered [ 1856.555592] Key type ._llcrypt registered [ 1856.562202] Key type .llcrypt registered [ 1856.700169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1856.712394] alg: No test for adler32 (adler32-zlib) [ 1857.569248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1857.587426] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1857.609840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1863.537717] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:23:53 (1788798233) [ 1865.003682] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1866.187860] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:23:56 (1788798236) [ 1867.742684] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1869.386815] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:23:59 (1788798239) [ 1869.689858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1869.710630] LNet: 112994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.717491] LNetError: 112994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.731879] LNet: Removed LNI 192.168.206.49@tcp [ 1869.755648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1869.797420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1869.821684] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1870.495977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1870.529397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1877.728066] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:24:07 (1788798247) [ 1877.858163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1877.895595] LNet: 113701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.903241] LNetError: 113701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.933375] LNet: Removed LNI 192.168.206.49@tcp [ 1877.963700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.008894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1878.039316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1878.656349] Key type .llcrypt unregistered [ 1878.662154] Key type ._llcrypt unregistered [ 1890.596129] Key type ._llcrypt registered [ 1890.602182] Key type .llcrypt registered [ 1891.908237] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:24:21 (1788798261) [ 1892.346645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1892.366934] alg: No test for adler32 (adler32-zlib) [ 1893.281667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1893.353653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1893.378680] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1894.118521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1894.177879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1900.640111] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:24:30 (1788798270) [ 1900.789466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1900.810950] LNet: 115254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.821767] LNetError: 115254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.862857] LNet: Removed LNI 192.168.206.49@tcp [ 1901.915951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1902.033976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1902.104935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1908.434158] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:24:38 (1788798278) [ 1908.723589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1908.750056] LNet: 115837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.780446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1908.863717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1909.325200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1909.359795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1909.413780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1909.781955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1909.823785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1909.905957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.920669] LNet: 115870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.935756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.985181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1910.385204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1910.413840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1910.492856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1910.919105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1910.959163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1917.603809] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:24:47 (1788798287) [ 1917.980834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1918.006892] LNet: 116470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1918.058890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.137951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1918.157914] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1918.870450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1918.892169] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 1919.684630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1919.742349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1919.993861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1920.025215] LNetError: 116750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.032290] LNet: Removed LNI 192.168.206.49@tcp [ 1920.041246] LNet: Removed LNI 192.168.206.49@tcp2 [ 1920.059802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.149802] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1920.156944] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 1920.164556] LNet: 116756:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1920.910299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1921.167670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1921.199303] LNetError: 116793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1921.211069] LNetError: 116793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1921.217078] LNet: Removed LNI 192.168.206.49@tcp [ 1921.238634] LNet: Removed LNI 192.168.206.49@tcp2 [ 1921.288540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.362721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1921.388650] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1922.068331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1922.091728] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 1922.753764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1922.805217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1922.854837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1923.085950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1923.102006] LNet: 117077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1923.106168] LNet: 117077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1924.127171] LNet: Removed LNI 192.168.206.49@tcp [ 1924.133197] LNetError: 117077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1924.146712] LNetError: 117077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1924.165592] LNet: Removed LNI 192.168.206.49@tcp2 [ 1924.184449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1924.247632] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1924.258965] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 1924.264456] LNet: 117084:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1924.713808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1924.962297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1924.985677] LNet: Removed LNI 192.168.206.49@tcp [ 1926.053941] LNet: Removed LNI 192.168.206.49@tcp2 [ 1926.090648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1926.178398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1926.223062] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1927.034995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1927.064369] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 1928.462970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1928.534615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1928.817593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1928.872504] LNetError: 117400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.886098] LNetError: 117400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1928.897279] LNet: Removed LNI 192.168.206.49@tcp [ 1928.925150] LNet: Removed LNI 192.168.206.49@tcp2 [ 1928.984518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.118355] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1929.133163] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 1929.146036] LNet: 117406:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1929.774501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1929.960211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1931.056482] LNet: Removed LNI 192.168.206.49@tcp [ 1931.080845] LNet: Removed LNI 192.168.206.49@tcp2 [ 1931.116557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1931.194789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1931.232778] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1932.046352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1932.078496] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 1932.953764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1932.996805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1933.088859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1933.305896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1933.346612] LNet: 117728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.353538] LNet: 117728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1934.367186] LNet: Removed LNI 192.168.206.49@tcp [ 1935.409045] LNet: Removed LNI 192.168.206.49@tcp2 [ 1935.494795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1935.612429] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1935.626985] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 1935.635615] LNet: 117735:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1936.121978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1936.280230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1936.298385] LNet: Removed LNI 192.168.206.49@tcp [ 1937.323572] LNet: Removed LNI 192.168.206.49@tcp2 [ 1937.385056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1937.505486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1937.560845] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1938.239484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1938.272395] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 1939.505773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1939.575078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1939.691464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1940.128948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1940.156429] LNetError: 118069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1940.164986] LNetError: 118069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1941.216946] LNet: Removed LNI 192.168.206.49@tcp [ 1941.247962] LNet: Removed LNI 192.168.206.49@tcp2 [ 1941.293545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.397242] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1941.416296] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 1941.422586] LNet: 118076:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1942.103352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1942.487398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.519287] LNet: Removed LNI 192.168.206.49@tcp [ 1943.531794] LNet: Removed LNI 192.168.206.49@tcp2 [ 1943.561544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.640700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1943.677554] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1944.299699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1944.316901] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 1945.007808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1945.046549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1945.078597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1945.153907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1945.516477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1946.591616] LNet: Removed LNI 192.168.206.49@tcp [ 1947.625921] LNet: Removed LNI 192.168.206.49@tcp2 [ 1947.668709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.754955] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1947.767868] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 1948.405405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1955.493347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1956.986035] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:25:26 (1788798326) [ 1957.075324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1957.103248] LNet: 119190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1957.111072] LNet: 119190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1957.128956] LNetError: 119190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1957.141622] LNetError: 119190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1957.150275] LNet: Removed LNI 192.168.206.49@tcp [ 1958.186864] LNet: Removed LNI 192.168.206.49@tcp2 [ 1958.245850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.563111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1958.604986] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1959.196403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1959.216622] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1959.991967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1960.023478] LNet: Removed LNI 10.1.2.3@tcp [ 1960.039152] LNet: Removed LNI 192.168.206.49@tcp [ 1960.062183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1960.121648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1960.154738] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1960.160570] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1960.626525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1960.653803] LNet: Removed LNI 192.168.206.49@tcp [ 1960.664965] LNet: Removed LNI 10.1.2.3@tcp [ 1966.516201] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 12:25:36 (1788798336) [ 1966.570837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1966.628570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1966.704724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1966.733766] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1967.545676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1967.574197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1967.626224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1967.663165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1967.689205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1967.718712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1967.752420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1967.783671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1967.818812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1967.855479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1967.894841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1967.922340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1968.060087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1973.930423] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 12:25:43 (1788798343) [ 1974.038092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1975.081965] LNet: Removed LNI 192.168.206.49@tcp [ 1975.109972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.187660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1975.213577] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1975.807691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 1975.859703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1975.919230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1975.950531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1975.981243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.009194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.060772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.086906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.127492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.185725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.224660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.261318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.362723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1976.398702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.455807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.499719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.554847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.619900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.661837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.684554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.707450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.730345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1976.756891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.49@tcp [ 1982.457169] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:25:52 (1788798352) [ 1982.733804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1983.786536] LNet: Removed LNI 192.168.206.49@tcp [ 1983.839162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1984.057657] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1984.301682] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1984.304977] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1984.307830] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1986.663943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1986.688591] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 1987.677687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1987.698660] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1989.630442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 1989.688187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1989.779184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.49@tcp -m PUT -e local_timeout -r 1 [ 1989.833943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1989.902712] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1989.904731] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1990.172251] LNetError: 121633:0:(lib-move.c:4794:lnet_parse()) 192.168.206.49@tcp, src 192.168.206.49@tcp: Dropping PUT (error -108 looking up sender) [ 1991.206935] LNet: 122092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.221825] LNet: 122092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1991.229188] LNetError: 122092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1991.242834] LNetError: 122092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1991.252937] LNet: Removed LNI 10.1.2.3@tcp [ 1992.321567] LNet: Removed LNI 192.168.206.49@tcp [ 1992.868217] Key type .llcrypt unregistered [ 1992.877302] Key type ._llcrypt unregistered [ 2003.728057] Key type ._llcrypt registered [ 2003.731605] Key type .llcrypt registered [ 2005.101481] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:26:14 (1788798374) [ 2005.443337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2005.454844] alg: No test for adler32 (adler32-zlib) [ 2006.320611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.727778] LNet: 122943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.754820] Key type .llcrypt unregistered [ 2007.759175] Key type ._llcrypt unregistered [ 2016.240928] Key type ._llcrypt registered [ 2016.248324] Key type .llcrypt registered [ 2016.497510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2016.516398] alg: No test for adler32 (adler32-zlib) [ 2017.529339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2017.571232] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 2026.973649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.149@tcp [ 2027.241159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2027.883868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.149@tcp [ 2028.277184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2028.328618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2028.381199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2028.422069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2028.476597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2028.523625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2028.585967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2028.650114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2028.703916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2028.747468] LNet: 123393:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2028.779336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2028.824208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2028.834491] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.149@tcp [ 2028.848108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2028.868197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2028.888354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2028.915318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2029.637918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2029.861164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2029.917970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2029.964478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2030.020673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2030.051523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2030.082697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2030.114655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2030.161621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2030.205157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2030.230894] LNet: 123512:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2030.262510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2030.298582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2030.347500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2030.398237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2030.423598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2030.444418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2031.404118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.149@tcp [ 2031.746618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2031.786303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2031.838448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2031.879268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2031.904848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2031.929775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2031.970704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2031.997290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2032.017641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2032.031248] LNet: 123632:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2032.046226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2032.065887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2032.086473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2032.107941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2032.134986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2032.182849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2032.791320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2032.985479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2033.036387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2033.096820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2033.131801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2033.193951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2033.220323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2033.250113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2033.285750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2033.363894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2033.428975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2033.472679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2033.517452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2033.562366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2033.601304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2033.637557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2033.980931] LNet: 123797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.988918] LNetError: 123797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2034.008509] LNet: Removed LNI 192.168.206.49@tcp [ 2034.439833] Key type .llcrypt unregistered [ 2034.441315] Key type ._llcrypt unregistered [ 2049.946564] Key type ._llcrypt registered [ 2049.949034] Key type .llcrypt registered [ 2051.158941] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:27:00 (1788798420) [ 2051.601237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2051.625436] alg: No test for adler32 (adler32-zlib) [ 2052.584029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2052.923693] LNet: 124681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.564248] Key type .llcrypt unregistered [ 2053.569077] Key type ._llcrypt unregistered [ 2059.708688] Key type ._llcrypt registered [ 2059.710806] Key type .llcrypt registered [ 2059.939289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2059.974208] alg: No test for adler32 (adler32-zlib) [ 2060.904327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2060.955136] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 2068.631428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.149@tcp [ 2076.103899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.145339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2076.162846] LNet: Added LNI 192.168.206.49@tcp1 [8/256/0/180] [ 2077.108733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2077.779698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.149@tcp [ 2078.034959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2078.070158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2078.098809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2078.119086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2078.140506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2078.179058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2078.210758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2078.246233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2078.281468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2078.309705] LNet: 125332:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2078.329738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2078.362869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2078.368240] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.149@tcp [ 2078.395795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2078.434918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2078.490756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2078.536029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2079.070556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2079.269393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2079.296652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2079.324213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2079.364482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2079.403928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2079.435549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2079.468573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2079.498854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2079.542155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2079.561667] LNet: 125451:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2079.580738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2079.608118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2079.646162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2079.676681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2079.707715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2079.738188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2080.321284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.149@tcp [ 2080.534350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2080.573749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2080.606116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2080.638269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2080.686907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2080.740697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2080.778465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2080.807861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2080.829589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2080.845688] LNet: 125570:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2080.862442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2080.901614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2080.947599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2080.981129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2081.009963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2081.057910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2081.745477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2082.021159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2082.054944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2082.087781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2082.142958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2082.197971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2082.244974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2082.290722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2082.331302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2082.372974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2082.399878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2082.430481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2082.481442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2082.513274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2082.547276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2082.586523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2087.048761] LNet: 125770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.057727] LNetError: 125770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2087.068690] LNet: Removed LNI 192.168.206.49@tcp [ 2087.080841] LNet: Removed LNI 192.168.206.49@tcp1 [ 2087.523961] Key type .llcrypt unregistered [ 2087.528632] Key type ._llcrypt unregistered [ 2102.967666] Key type ._llcrypt registered [ 2102.972381] Key type .llcrypt registered [ 2104.225794] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 12:27:53 (1788798473) [ 2104.611769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2104.627058] alg: No test for adler32 (adler32-zlib) [ 2105.520583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.579996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2105.626989] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 2106.292852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2106.336443] LNet: Added LNI 192.168.206.49@tcp2 [8/256/0/180] [ 2107.066853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.49@tcp [ 2107.145722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2108.194052] LNet: Removed LNI 192.168.206.49@tcp [ 2113.755855] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 12:28:03 (1788798483) [ 2115.056430] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2116.681492] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:28:06 (1788798486) [ 2122.128507] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:28:12 (1788798492) [ 2127.989356] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:28:17 (1788798497) [ 2134.612142] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:28:24 (1788798504) [ 2140.126502] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:28:30 (1788798510) [ 2145.579627] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:28:35 (1788798515) [ 2152.072785] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:28:41 (1788798521) [ 2159.997102] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:28:49 (1788798529) [ 2167.673943] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:28:57 (1788798537) [ 2173.552160] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:29:03 (1788798543) [ 2179.087336] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:29:09 (1788798549) [ 2185.929202] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:29:15 (1788798555) [ 2191.815424] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:29:21 (1788798561) [ 2197.897634] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:29:27 (1788798567) [ 2202.840085] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:29:32 (1788798572) [ 2208.638416] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:29:38 (1788798578) [ 2215.708995] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:29:45 (1788798585) [ 2221.272200] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:29:51 (1788798591) [ 2221.497247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2221.521716] LNet: 137458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2221.530718] LNetError: 137458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2221.550650] LNet: Removed LNI 192.168.206.49@tcp2 [ 2221.572619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2221.691732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2221.726927] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 2222.618254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2222.638786] LNet: 137600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.646760] LNetError: 137600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.666172] LNet: Removed LNI 192.168.206.49@tcp [ 2222.701496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.743325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2222.761828] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 2228.124343] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:29:58 (1788798598) [ 2234.898311] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:30:04 (1788798604) [ 2242.189300] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:30:11 (1788798611) [ 2248.160627] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:30:18 (1788798618) [ 2253.570977] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:30:23 (1788798623) [ 2259.681758] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:30:29 (1788798629) [ 2264.787307] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:30:34 (1788798634) [ 2270.056889] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:30:40 (1788798640) [ 2276.423126] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:30:46 (1788798646) [ 2283.494586] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:30:53 (1788798653) [ 2289.923845] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:30:59 (1788798659) [ 2296.831410] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:31:06 (1788798666) [ 2303.133339] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2080 sec ========== 12:31:12 (1788798672) [ 2303.550202] LNet: 145063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2303.561921] LNetError: 145063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2303.604150] LNet: Removed LNI 192.168.206.49@tcp [ 2304.223184] Key type .llcrypt unregistered [ 2304.230630] Key type ._llcrypt unregistered [ 2315.820695] Key type ._llcrypt registered [ 2315.822905] Key type .llcrypt registered [ 2316.234519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2316.245772] alg: No test for adler32 (adler32-zlib) [ 2317.249835] Lustre: Lustre: Build Version: 2.17.58_39_g569731d [ 2317.522555] LNet: Added LNI 192.168.206.49@tcp [8/256/0/180] [ 2319.207250] Key type lgssc registered [ 2320.369520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2381.410075] Lustre: Mounted lustre-client - version 2.17.58_39_g569731d [ 2387.068643] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2394.077781] Lustre: Unmounted lustre-client [ 2419.893239] Key type lgssc unregistered [ 2420.240939] LNet: 148007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2420.255650] LNetError: 148007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.274678] LNet: Removed LNI 192.168.206.49@tcp [ 2420.953154] Key type .llcrypt unregistered [ 2420.957979] Key type ._llcrypt unregistered