[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 447777885 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003368] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.006010] kvm-guest: setup PV IPIs [ 0.009321] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011012] pid_max: default: 32768 minimum: 301 [ 0.012124] LSM: Security Framework initializing [ 0.013049] Yama: becoming mindful. [ 0.014036] SELinux: Initializing. [ 0.016072] *** VALIDATE selinux *** [ 0.024344] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029491] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031103] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032103] *** VALIDATE tmpfs *** [ 0.033416] *** VALIDATE proc *** [ 0.034201] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.036243] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038066] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041007] Spectre V2 : User space: Vulnerable [ 0.042010] Speculative Store Bypass: Vulnerable [ 0.044630] debug: unmapping init [mem 0xffffffffbce59000-0xffffffffbce60fff] [ 0.047167] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048810] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049022] ... version: 2 [ 0.050016] ... bit width: 48 [ 0.051010] ... generic registers: 4 [ 0.052012] ... value mask: 0000ffffffffffff [ 0.053014] ... max period: 00007fffffffffff [ 0.054016] ... fixed-purpose events: 3 [ 0.055017] ... event mask: 000000070000000f [ 0.056307] rcu: Hierarchical SRCU implementation. [ 0.058467] smp: Bringing up secondary CPUs ... [ 0.059595] x86: Booting SMP configuration: [ 0.060054] .... node #0, CPUs: #1 #2 #3 [ 0.063396] smp: Brought up 1 node, 4 CPUs [ 0.065012] smpboot: Max logical packages: 1 [ 0.066019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.276160] node 0 deferred pages initialised in 207ms [ 0.280012] devtmpfs: initialized [ 0.281222] x86/mm: Memory block size: 128MB [ 0.283659] gcov: version magic: 0x41383552 [ 0.285133] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.286077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.287235] pinctrl core: initialized pinctrl subsystem [ 0.288173] [ 0.288792] ************************************************************* [ 0.289013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.290012] ** ** [ 0.291014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.292013] ** ** [ 0.293018] ** This means that this kernel is built to expose internal ** [ 0.294014] ** IOMMU data structures, which may compromise security on ** [ 0.295014] ** your system. ** [ 0.296014] ** ** [ 0.297016] ** If you see this message and you are not debugging the ** [ 0.298021] ** kernel, report this immediately to your vendor! ** [ 0.299013] ** ** [ 0.300014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.301018] ************************************************************* [ 0.302574] NET: Registered protocol family 16 [ 0.303420] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.304057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.305059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.306369] cpuidle: using governor menu [ 0.309528] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.312659] PCI: Using configuration type 1 for base access [ 0.314192] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.324110] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.325037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.327076] cryptd: max_cpu_qlen set to 1000 [ 0.330394] ACPI: Added _OSI(Module Device) [ 0.332028] ACPI: Added _OSI(Processor Device) [ 0.334021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.336024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.341586] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.348597] ACPI: Interpreter enabled [ 0.351086] ACPI: PM: (supports S0 S3 S4 S5) [ 0.352018] ACPI: Using IOAPIC for interrupt routing [ 0.354152] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.358477] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.369287] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.371053] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.374026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.378105] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.383535] acpiphp: Slot [2] registered [ 0.385164] acpiphp: Slot [5] registered [ 0.386137] acpiphp: Slot [6] registered [ 0.388126] acpiphp: Slot [3] registered [ 0.389102] acpiphp: Slot [4] registered [ 0.391112] acpiphp: Slot [7] registered [ 0.393098] acpiphp: Slot [8] registered [ 0.394307] acpiphp: Slot [9] registered [ 0.396102] acpiphp: Slot [10] registered [ 0.398157] acpiphp: Slot [11] registered [ 0.399096] acpiphp: Slot [12] registered [ 0.401106] acpiphp: Slot [13] registered [ 0.403118] acpiphp: Slot [14] registered [ 0.404096] acpiphp: Slot [15] registered [ 0.406116] acpiphp: Slot [16] registered [ 0.407126] acpiphp: Slot [17] registered [ 0.409105] acpiphp: Slot [18] registered [ 0.410087] acpiphp: Slot [19] registered [ 0.411100] acpiphp: Slot [20] registered [ 0.413139] acpiphp: Slot [21] registered [ 0.415124] acpiphp: Slot [22] registered [ 0.416123] acpiphp: Slot [23] registered [ 0.418104] acpiphp: Slot [24] registered [ 0.419121] acpiphp: Slot [25] registered [ 0.421120] acpiphp: Slot [26] registered [ 0.423127] acpiphp: Slot [27] registered [ 0.425126] acpiphp: Slot [28] registered [ 0.426132] acpiphp: Slot [29] registered [ 0.427108] acpiphp: Slot [30] registered [ 0.428095] acpiphp: Slot [31] registered [ 0.430065] PCI host bridge to bus 0000:00 [ 0.431022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.433027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.434026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.437028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.438027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.441030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.442178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.445025] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.447255] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.454564] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.458062] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.461025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.463019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.465034] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.466621] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.469608] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.473048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.476940] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.482026] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.492807] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.499013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.505000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.513021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.519019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.536016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.544803] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.554982] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.559924] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.570920] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.581368] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.583414] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.585435] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.588500] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.591261] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.595184] iommu: Default domain type: Passthrough [ 0.597411] SCSI subsystem initialized [ 0.598138] ACPI: bus type USB registered [ 0.600121] usbcore: registered new interface driver usbfs [ 0.601069] usbcore: registered new interface driver hub [ 0.603080] usbcore: registered new device driver usb [ 0.604176] pps_core: LinuxPPS API ver. 1 registered [ 0.606010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.608059] PTP clock support registered [ 0.610079] EDAC MC: Ver: 3.0.0 [ 0.612149] PCI: Using ACPI for IRQ routing [ 0.613523] NetLabel: Initializing [ 0.615013] NetLabel: domain hash size = 128 [ 0.616008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.618078] NetLabel: unlabeled traffic allowed by default [ 0.620045] vgaarb: loaded [ 0.621345] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.623023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.628559] clocksource: Switched to clocksource kvm-clock [ 0.737755] VFS: Disk quotas dquot_6.6.0 [ 0.738832] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.741045] *** VALIDATE ramfs *** [ 0.741858] *** VALIDATE hugetlbfs *** [ 0.743273] pnp: PnP ACPI init [ 0.745347] pnp: PnP ACPI: found 6 devices [ 0.767326] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.769877] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.772110] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.774355] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.776289] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.778626] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.781483] NET: Registered protocol family 2 [ 0.783992] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.788678] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.791701] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.797274] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.801107] TCP: Hash tables configured (established 65536 bind 65536) [ 0.804264] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.807657] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.810775] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.814144] NET: Registered protocol family 1 [ 0.817129] RPC: Registered named UNIX socket transport module. [ 0.818837] RPC: Registered udp transport module. [ 0.820217] RPC: Registered tcp transport module. [ 0.821529] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.823382] NET: Registered protocol family 44 [ 0.824626] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.826363] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.827902] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.829681] PCI: CLS 0 bytes, default 64 [ 0.831034] Unpacking initramfs... [ 2.241576] debug: unmapping init [mem 0xffff92117cc64000-0xffff92117ffcffff] [ 2.245527] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.247751] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.249823] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.767270] Initialise system trusted keyrings [ 2.771933] Key type blacklist registered [ 2.773888] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.784428] zbud: loaded [ 2.787708] *** VALIDATE nfs *** [ 2.789273] *** VALIDATE nfs4 *** [ 2.790859] pstore: using deflate compression [ 2.795087] Platform Keyring initialized [ 2.895556] NET: Registered protocol family 38 [ 2.897538] Key type asymmetric registered [ 2.898966] Asymmetric key parser 'x509' registered [ 2.900830] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.903485] io scheduler mq-deadline registered [ 2.905470] io scheduler kyber registered [ 2.907351] io scheduler bfq registered [ 2.909358] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.912812] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.916046] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.919078] ACPI: Power Button [PWRF] [ 2.926131] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.932799] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.945802] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.974529] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.004940] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.011478] Non-volatile memory driver v1.3 [ 3.014127] Linux agpgart interface v0.103 [ 3.047680] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.051570] vda: detected capacity change from 0 to 74829824 [ 3.067377] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.069590] vdb: detected capacity change from 0 to 1073741824 [ 3.078746] libphy: Fixed MDIO Bus: probed [ 3.083996] usbcore: registered new interface driver usbserial_generic [ 3.086753] usbserial: USB Serial support registered for generic [ 3.089430] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.094767] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.096794] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.099567] mousedev: PS/2 mouse device common for all mice [ 3.103394] rtc_cmos 00:05: RTC can wake from S4 [ 3.106253] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.107639] rtc_cmos 00:05: registered as rtc0 [ 3.112624] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.113674] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.118538] intel_pstate: CPU model not supported [ 3.124454] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.128864] hid: raw HID events driver (C) Jiri Kosina [ 3.130824] usbcore: registered new interface driver usbhid [ 3.133150] usbhid: USB HID core driver [ 3.134611] drop_monitor: Initializing network drop monitor service [ 3.136830] Initializing XFRM netlink socket [ 3.138721] NET: Registered protocol family 10 [ 3.141796] Segment Routing with IPv6 [ 3.143519] NET: Registered protocol family 17 [ 3.146082] mpls_gso: MPLS GSO support [ 3.152717] RAS: Correctable Errors collector initialized. [ 3.155373] AVX version of gcm_enc/dec engaged. [ 3.156828] AES CTR mode by8 optimization enabled [ 3.234459] sched_clock: Marking stable (3234432494, 0)->(4141540454, -907107960) [ 3.238546] registered taskstats version 1 [ 3.240377] Loading compiled-in X.509 certificates [ 3.243031] zswap: loaded using pool lzo/zbud [ 3.269868] Key type big_key registered [ 3.283687] Key type encrypted registered [ 3.285744] ima: No TPM chip found, activating TPM-bypass! [ 3.288380] ima: Allocated hash algorithm: sha1 [ 3.290384] ima: No architecture policies found [ 3.292389] evm: Initialising EVM extended attributes: [ 3.294475] evm: security.selinux [ 3.295845] evm: security.ima [ 3.297123] evm: security.capability [ 3.298657] evm: HMAC attrs: 0x1 [ 3.301442] rtc_cmos 00:05: setting system clock to 2026-08-24 12:21:29 UTC (1787574089) [ 3.308557] debug: unmapping init [mem 0xffffffffbde03000-0xffffffffbdffffff] [ 3.311943] debug: unmapping init [mem 0xffffffffbcb82000-0xffffffffbce58fff] [ 3.321174] Write protecting the kernel read-only data: 28672k [ 3.324921] debug: unmapping init [mem 0xffffffffbb203000-0xffffffffbb3fffff] [ 3.327932] debug: unmapping init [mem 0xffffffffbbb14000-0xffffffffbbbfffff] [ 3.361483] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.368601] systemd[1]: Detected virtualization kvm. [ 3.370507] systemd[1]: Detected architecture x86-64. [ 3.372716] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.405051] systemd[1]: No hostname configured. [ 3.407089] systemd[1]: Set hostname to . [ 3.409429] random: systemd: uninitialized urandom read (16 bytes read) [ 3.411992] systemd[1]: Initializing machine ID from random generator. [ 3.456962] random: ln: uninitialized urandom read (6 bytes read) [ 3.541805] random: systemd: uninitialized urandom read (16 bytes read) [ 3.544386] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.549696] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.556274] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.145198] device-mapper: uevent: version 1.0.3 [ 4.147594] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.910866] random: fast init done [ 4.979333] virtio_net virtio0 ens2: renamed from eth0 [ 5.010391] scsi host0: ata_piix [ 5.040099] scsi host1: ata_piix [ 5.043403] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.045707] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.786713] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.804226] random: crng init done [ 9.805906] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.225395] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.333422] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.594993] SELinux: Disabled at runtime. [ 11.657592] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.666792] systemd[1]: Detected virtualization kvm. [ 11.668995] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.188913] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.192210] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.195688] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.199266] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.202125] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.211949] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.219434] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 12.259153] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ 12.601932] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.896394] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.986101] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.055870] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.067795] EDAC sbridge: Ver: 1.1.2 [ 14.022332] Key type dns_resolver registered [ 14.319452] NFS: Registering the id_resolver key type [ 14.322033] Key type id_resolver registered [ 14.323701] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg601-client login: [ 59.928875] libcfs: loading out-of-tree module taints kernel. [ 60.111834] Key type ._llcrypt registered [ 60.116809] Key type .llcrypt registered [ 61.128492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 61.154408] alg: No test for adler32 (adler32-zlib) [ 62.886603] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 64.305897] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 66.104537] Key type lgssc registered [ 68.023186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 190.482404] Lustre: Mounted lustre-client - version 2.17.57_80_g1214e44 [ 196.858379] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 210.542437] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing check_logdir /tmp/testlogs/ [ 215.470389] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing yml_node [ 216.034221] Lustre: lustre-OST0000-osc-ffff9211c72df800: disconnect after 23s idle [ 219.112108] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 221.412063] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 223.826837] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 225.104171] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 24 08:25:10 EDT 2026 [ 240.070460] Lustre: Unmounted lustre-client [ 267.610146] Key type lgssc unregistered [ 268.008665] LNet: 8414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 268.025600] LNetError: 8414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 268.044850] LNet: Removed LNI 192.168.206.1@tcp [ 268.983170] Key type .llcrypt unregistered [ 268.985318] Key type ._llcrypt unregistered [ 294.025895] Key type ._llcrypt registered [ 294.030500] Key type .llcrypt registered [ 294.316503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 294.346183] alg: No test for adler32 (adler32-zlib) [ 295.387654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 295.432579] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 295.450596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 299.426902] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 299.530909] LNet: 9293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.544981] LNetError: 9293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 299.571546] LNet: Removed LNI 192.168.206.1@tcp [ 300.537495] Key type .llcrypt unregistered [ 300.540550] Key type ._llcrypt unregistered [ 306.598793] Key type ._llcrypt registered [ 306.602336] Key type .llcrypt registered [ 307.963134] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 08:26:32 (1787574392) [ 308.686878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 308.708188] alg: No test for adler32 (adler32-zlib) [ 309.604502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.723501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.765656] LNet: 9716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.793882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.838655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 309.856572] LNet: 9723:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 315.880212] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 08:26:41 (1787574401) [ 315.977708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.028165] LNet: 10296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.073272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.149295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 322.128320] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 08:26:46 (1787574406) [ 322.202921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.229788] LNet: 10880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.254369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.367955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 328.128389] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 08:26:53 (1787574413) [ 328.208192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.221879] LNet: 11465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.241500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.342660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 333.724348] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 08:26:58 (1787574418) [ 333.761824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.778427] LNet: 12047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.792149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.864390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 334.218642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 339.288442] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 08:27:04 (1787574424) [ 339.411414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.456568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.515964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 340.013500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 340.387557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 345.822881] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 08:27:11 (1787574431) [ 345.988113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.031915] LNet: 13241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.037845] LNet: 13241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 346.056434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.126779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 351.054773] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 08:27:16 (1787574436) [ 351.105826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.135790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.186675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 351.214964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 351.501695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 351.540621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 351.951266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 351.996701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 352.337175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 352.381528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 352.666248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 352.689863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 352.911334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 352.931238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 353.171943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 353.195420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 353.437797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 353.461727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 353.809963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 353.857946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 354.132952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 354.163607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 354.526333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 354.565924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 354.958560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 354.995724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 355.400520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 355.461512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 361.378829] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 08:27:26 (1787574446) [ 361.640646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.672597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.738411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 361.768170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 367.960671] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 08:27:32 (1787574452) [ 368.062463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.088154] LNet: 15089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.095960] LNet: 15089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 368.137332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.229735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 368.271910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 373.749949] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 08:27:38 (1787574458) [ 373.827420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.887539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.983778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 374.019964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 380.001175] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 08:27:44 (1787574464) [ 380.080881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.119612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.204769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 380.245352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 385.987234] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 08:27:51 (1787574471) [ 386.314758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.382667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.517182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 386.587298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 394.709884] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 08:27:58 (1787574478) [ 394.837984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.885156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.033421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 395.116087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 402.553825] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 08:28:07 (1787574487) [ 402.835752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.866273] LNet: 18028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.873961] LNet: 18028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 402.908158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.017360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 403.535832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 404.402725] hrtimer: interrupt took 2813758 ns [ 410.718769] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 08:28:15 (1787574495) [ 410.839414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.867351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.077648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 411.386530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 419.617516] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 08:28:24 (1787574504) [ 419.727706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.779891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.964612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 420.207400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 427.628528] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 08:28:32 (1787574512) [ 427.730703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.802273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.500316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.809909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 435.806462] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 08:28:40 (1787574520) [ 435.911199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.977887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.033241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 436.066974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 442.979846] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 08:28:47 (1787574527) [ 443.173380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.248705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.354196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 443.408465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 450.667759] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 08:28:55 (1787574535) [ 450.779700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.828256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.001978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 458.272116] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 08:29:02 (1787574542) [ 458.366210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.414438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.523677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 464.363799] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 08:29:09 (1787574549) [ 464.482696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.563214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.696574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 465.145564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 471.737725] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 08:29:16 (1787574556) [ 471.865151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.901885] LNet: 23463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 471.909301] LNet: 23463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 471.947502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.059794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 472.089436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 479.032633] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 08:29:23 (1787574563) [ 479.166912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.212919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.301489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 479.331046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 486.202512] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 08:29:30 (1787574570) [ 486.565642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.617401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.694855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 486.772837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 495.313766] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 08:29:39 (1787574579) [ 495.484563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.550394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.653632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 495.694785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 503.818897] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 08:29:48 (1787574588) [ 503.894580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.972531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.063452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 504.132638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 504.154432] LNetError: 25827:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 504.193876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 504.231312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 510.722149] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 08:29:55 (1787574595) [ 510.930988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.988578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.051542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.S55r9alMkn [ 511.111331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.S55r9alMkn [ 511.149891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.S55r9alMkn [ 516.809820] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 08:30:01 (1787574601) [ 517.048967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.094365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.157232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 517.183850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 517.230706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 523.134573] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 08:30:07 (1787574607) [ 523.231225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.276154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.332916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 523.369306] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 524.984451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 526.568410] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 528.632986] Key type lgssc registered [ 530.260864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 535.777837] LNetError: 27586:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.1@tcp99 has no available nets [ 539.920394] Key type lgssc unregistered [ 540.483425] LNetError: 28701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 541.542704] LNet: Removed LNI 192.168.206.1@tcp [ 542.368313] Key type .llcrypt unregistered [ 542.373755] Key type ._llcrypt unregistered [ 550.598653] Key type ._llcrypt registered [ 550.603630] Key type .llcrypt registered [ 551.869342] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 08:30:36 (1787574636) [ 552.777639] Key type .llcrypt unregistered [ 552.782115] Key type ._llcrypt unregistered [ 558.025326] Key type ._llcrypt registered [ 558.027532] Key type .llcrypt registered [ 558.277847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 558.317101] alg: No test for adler32 (adler32-zlib) [ 559.286307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.358291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 559.369631] LNet: 29622:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 559.378582] LNet: 29622:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 559.389244] LNet: 29622:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 559.405651] LNet: Message forwarding will be enabled in 2s [ 562.464192] LNet: Message forwarding enabled [ 563.024642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 563.080139] LNet: Message forwarding will stop in 2s to 6s [ 566.564283] LNet: Message forwarding disabled [ 567.744995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 567.775140] LNet: 29720:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 567.791364] LNet: 29720:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 567.811529] LNet: 29720:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 567.841702] LNet: Message forwarding will be enabled in 2s [ 568.014625] LNet: 29733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 568.616539] Key type .llcrypt unregistered [ 568.623520] Key type ._llcrypt unregistered [ 578.355729] Key type ._llcrypt registered [ 578.358174] Key type .llcrypt registered [ 579.371506] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 08:31:04 (1787574664) [ 579.694948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 579.706535] alg: No test for adler32 (adler32-zlib) [ 580.609201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.651994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 586.069584] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 08:31:11 (1787574671) [ 586.385083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.408139] LNet: 31141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.425317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.475149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 586.507113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 586.529674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 586.566145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 586.593633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 586.630517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 586.669922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 586.698808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 586.729066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 586.765223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 586.805760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 586.835741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 586.881407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 586.916620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 586.948546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 586.993977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 587.031657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 587.080928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 587.122618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 587.140708] LNetError: 31184:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 592.559646] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 08:31:17 (1787574677) [ 592.662721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.683970] LNet: 31749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.713943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.810611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 598.271370] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 08:31:23 (1787574683) [ 599.866599] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg601-server [ 601.309495] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 08:31:26 (1787574686) [ 601.466559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.493328] LNet: 32685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.509479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.559715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 601.575445] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 602.295341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[2-6]@tcp [ 602.310530] LNet: 32818:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 602.340685] LNetError: 32689:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.2@tcp is being used as a gateway but routing feature is not turned on [ 602.362700] LNetError: 32689:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.2@tcp has gone from up to down [ 607.383736] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 08:31:32 (1787574692) [ 608.829421] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg601-server [ 610.957294] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 08:31:35 (1787574695) [ 611.056776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.097397] LNet: 33747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.108291] LNetError: 33747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 611.134624] LNet: Removed LNI 192.168.206.1@tcp [ 611.172401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.252452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 611.275032] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 612.464332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[2-7/2]@tcp [ 612.497561] LNet: 33871:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 612.512839] LNet: 33871:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 612.540332] LNetError: 33758:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.6@tcp is being used as a gateway but routing feature is not turned on [ 612.544725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[2-7/2]@tcp [ 612.556065] LNetError: 33758:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 612.556099] LNetError: 33758:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.6@tcp has gone from up to down [ 612.556105] LNetError: 33758:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 620.056532] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 08:31:44 (1787574704) [ 620.235447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.273189] LNet: 34460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 620.286734] LNetError: 34460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 620.330738] LNet: Removed LNI 192.168.206.1@tcp [ 620.374238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.473587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 620.551525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.603934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.702460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 620.785939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.865273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.941810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 621.053661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.138979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.296664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 621.457773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.541944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.609699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 621.695920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.731491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.819125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 621.912480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.970914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.065862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 622.193291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.272614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.360464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 622.445435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.496499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.578808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 622.758740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.838676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.939968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.069700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.128818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.205222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 623.278336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.333640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.454735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.590631] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 08:31:56 (1787574716) [ 633.956959] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg601-server [ 636.315805] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 08:32:00 (1787574720) [ 639.140859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg601-server [ 640.946253] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 08:32:05 (1787574725) [ 641.206556] LNet: 35864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.213326] LNet: 35864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 641.898116] Key type .llcrypt unregistered [ 641.901313] Key type ._llcrypt unregistered [ 648.652948] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 648.860804] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 648.874581] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 648.881352] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 650.501274] Key type ._llcrypt registered [ 650.502777] Key type .llcrypt registered [ 650.731948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 650.751474] alg: No test for adler32 (adler32-zlib) [ 651.768812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.837506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 651.856704] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 652.496443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 652.519705] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 655.035764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 655.061440] LNet: Removed LNI 10.1.2.3@tcp [ 655.225622] LNet: 36580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.234470] LNetError: 36580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.252108] LNet: Removed LNI 192.168.206.1@tcp [ 656.052738] Key type .llcrypt unregistered [ 656.058904] Key type ._llcrypt unregistered [ 669.900437] Key type ._llcrypt registered [ 669.903313] Key type .llcrypt registered [ 671.611359] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 08:32:35 (1787574755) [ 672.323371] Key type .llcrypt unregistered [ 672.330588] Key type ._llcrypt unregistered [ 677.864171] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 677.910266] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 677.914839] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 677.918929] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 680.827519] Key type ._llcrypt registered [ 680.832698] Key type .llcrypt registered [ 681.021921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 681.033456] alg: No test for adler32 (adler32-zlib) [ 682.175858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.280118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 682.332908] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 683.692652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 683.723240] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 687.630278] LNet: 38081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.635690] LNetError: 38081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 687.647910] LNet: Removed LNI 10.1.2.3@tcp [ 687.667433] LNet: Removed LNI 192.168.206.1@tcp [ 688.618644] Key type .llcrypt unregistered [ 688.621308] Key type ._llcrypt unregistered [ 700.923617] Key type ._llcrypt registered [ 700.929084] Key type .llcrypt registered [ 702.183703] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 08:33:06 (1787574786) [ 702.800615] Key type .llcrypt unregistered [ 702.804816] Key type ._llcrypt unregistered [ 708.346029] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 708.499922] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 708.511844] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 708.520802] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 711.063486] Key type ._llcrypt registered [ 711.066499] Key type .llcrypt registered [ 711.223400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 711.241617] alg: No test for adler32 (adler32-zlib) [ 712.101842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.153199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 712.172371] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 713.837974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 713.858748] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 715.887275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 715.914250] LNet: Removed LNI 10.1.2.3@tcp [ 715.933299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 715.967387] LNet: Removed LNI 10.1.2.31@tcp [ 716.007653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 716.053400] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 716.761785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 716.780839] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 717.586757] LNet: 40007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.597595] LNetError: 40007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 717.606575] LNet: Removed LNI 10.1.2.31@tcp [ 717.619216] LNet: Removed LNI 10.1.2.3@tcp [ 718.302583] Key type .llcrypt unregistered [ 718.306547] Key type ._llcrypt unregistered [ 730.762460] Key type ._llcrypt registered [ 730.764619] Key type .llcrypt registered [ 732.035495] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 08:33:36 (1787574816) [ 732.471842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 732.500051] alg: No test for adler32 (adler32-zlib) [ 733.397363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.447656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 733.467190] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 734.106939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.121122] LNet: 40979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.123847] LNetError: 40979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.139145] LNet: Removed LNI 192.168.206.1@tcp [ 734.153273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.183264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.1@tcp [ 734.201224] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 734.782873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.1@tcp [ 734.810384] LNet: Removed LNI 192.168.206.1@tcp [ 734.858175] LNet: 41115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.464424] Key type .llcrypt unregistered [ 735.466050] Key type ._llcrypt unregistered [ 746.430271] Key type ._llcrypt registered [ 746.432946] Key type .llcrypt registered [ 748.038252] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 08:33:52 (1787574832) [ 750.318553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg601-server [ 752.078811] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 08:33:56 (1787574836) [ 752.645437] Key type .llcrypt unregistered [ 752.646909] Key type ._llcrypt unregistered [ 758.083025] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 758.259072] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 758.263287] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 758.271264] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 760.704461] Key type ._llcrypt registered [ 760.707112] Key type .llcrypt registered [ 760.916434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 760.938570] alg: No test for adler32 (adler32-zlib) [ 761.842887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.905465] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 762.662950] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 764.769507] LNet: 42957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.778202] LNetError: 42957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 764.784725] LNet: Removed LNI 192.168.206.1@tcp1 [ 764.800043] LNet: Removed LNI 10.1.2.3@tcp2 [ 765.386788] Key type .llcrypt unregistered [ 765.393687] Key type ._llcrypt unregistered [ 776.654491] Key type ._llcrypt registered [ 776.660761] Key type .llcrypt registered [ 778.216220] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 08:34:22 (1787574862) [ 778.596488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 778.612109] alg: No test for adler32 (adler32-zlib) [ 779.504492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.585177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 779.647769] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 780.409468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.432463] LNet: 43932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.438510] LNetError: 43932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.476100] LNet: Removed LNI 192.168.206.1@tcp [ 780.502230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.588598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 780.613460] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 786.387826] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 08:34:31 (1787574871) [ 786.546880] LNet: 44518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.552710] LNetError: 44518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.593111] LNet: Removed LNI 192.168.206.1@tcp [ 787.228533] Key type .llcrypt unregistered [ 787.234234] Key type ._llcrypt unregistered [ 792.243134] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 792.376937] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 792.384637] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 792.388140] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 794.858713] Key type ._llcrypt registered [ 794.863379] Key type .llcrypt registered [ 795.034286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 795.055354] alg: No test for adler32 (adler32-zlib) [ 795.961905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.032693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 796.063812] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 796.702502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 796.718920] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 798.617331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.634518] LNet: 45265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.639458] LNetError: 45265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 798.646917] LNet: Removed LNI 10.1.2.3@tcp [ 798.659488] LNet: Removed LNI 192.168.206.1@tcp [ 798.686424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.750968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 798.773118] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 798.777145] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 799.189614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.223510] LNet: 45292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.232733] LNetError: 45292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 799.243138] LNetError: 45292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 799.249873] LNet: Removed LNI 10.1.2.3@tcp [ 799.261325] LNet: Removed LNI 192.168.206.1@tcp [ 799.282184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.342860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 799.383726] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 799.387532] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.040846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 800.061217] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 800.858593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.894152] LNet: 45455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.899400] LNetError: 45455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 800.903155] LNetError: 45455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 800.908370] LNet: Removed LNI 10.1.2.3@tcp [ 800.910632] LNet: Removed LNI 192.168.206.1@tcp [ 800.919651] LNet: Removed LNI 192.168.206.1@tcp2 [ 800.944571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.997980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 801.022076] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 801.028420] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 801.034215] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 807.301419] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 08:34:52 (1787574892) [ 807.389728] LNet: 46044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.396801] LNetError: 46044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 807.403161] LNetError: 46044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 807.416176] LNet: Removed LNI 10.1.2.3@tcp [ 807.419944] LNet: Removed LNI 192.168.206.1@tcp [ 807.434174] LNet: Removed LNI 192.168.206.1@tcp2 [ 808.304153] Key type .llcrypt unregistered [ 808.310925] Key type ._llcrypt unregistered [ 813.977314] Key type ._llcrypt registered [ 813.980609] Key type .llcrypt registered [ 814.208959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 814.226712] alg: No test for adler32 (adler32-zlib) [ 815.172112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.236686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 815.270976] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 816.433872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.461716] LNet: 46452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.470239] LNetError: 46452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.488785] LNet: Removed LNI 192.168.206.1@tcp [ 816.517496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.597650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 816.627947] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 817.229792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.254618] LNet: 46482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.262190] LNetError: 46482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.283567] LNet: Removed LNI 192.168.206.1@tcp [ 817.314591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.395729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 817.433272] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 817.952598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.995893] LNet: Removed LNI 192.168.206.1@tcp [ 818.022188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.131994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 818.163698] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 818.681569] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 818.869890] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 818.875753] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 818.885842] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 821.271774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.288890] LNet: 46588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.297537] LNet: 46588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 821.302957] LNetError: 46588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.315416] LNetError: 46588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 821.328248] LNet: Removed LNI 192.168.206.1@tcp [ 821.350457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.429983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 821.470791] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 828.103701] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 08:35:13 (1787574913) [ 828.214803] LNet: 47176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.220046] LNetError: 47176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.245049] LNet: Removed LNI 192.168.206.1@tcp [ 828.860702] Key type .llcrypt unregistered [ 828.864928] Key type ._llcrypt unregistered [ 834.910841] Key type ._llcrypt registered [ 834.912307] Key type .llcrypt registered [ 835.114135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 835.133517] alg: No test for adler32 (adler32-zlib) [ 836.051663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.106666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 836.139974] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 836.888298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 836.905859] LNet: 47588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.911435] LNetError: 47588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.926252] LNet: Removed LNI 192.168.206.1@tcp [ 836.955526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.009940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 837.027971] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 842.934393] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 08:35:27 (1787574927) [ 843.195477] LNet: 48178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.214738] LNetError: 48178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.237543] LNet: Removed LNI 192.168.206.1@tcp [ 843.899521] Key type .llcrypt unregistered [ 843.902593] Key type ._llcrypt unregistered [ 849.721941] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 849.882963] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 849.897123] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 849.945541] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 852.612162] Key type ._llcrypt registered [ 852.614405] Key type .llcrypt registered [ 852.773635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 852.798681] alg: No test for adler32 (adler32-zlib) [ 853.743492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.812224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 853.847615] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 854.874551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 854.912476] LNet: 48633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.918963] LNetError: 48633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.935392] LNet: Removed LNI 192.168.206.1@tcp [ 854.964941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.071078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 855.102096] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 861.190161] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 08:35:46 (1787574946) [ 861.408408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.425725] LNet: 49231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.434970] LNetError: 49231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.451424] LNet: Removed LNI 192.168.206.1@tcp [ 861.486066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.587412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 866.180596] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 08:35:51 (1787574951) [ 866.327533] LNet: 49811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.885507] Key type .llcrypt unregistered [ 866.887253] Key type ._llcrypt unregistered [ 873.210852] Key type ._llcrypt registered [ 873.213950] Key type .llcrypt registered [ 873.429723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 873.477888] alg: No test for adler32 (adler32-zlib) [ 874.453495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.532914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 874.558730] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 875.279308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.295955] LNet: 50215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.300701] LNetError: 50215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.312883] LNet: Removed LNI 192.168.206.1@tcp [ 875.325818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.392399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 875.425346] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 881.342845] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 08:36:06 (1787574966) [ 881.671766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.694445] LNet: 50808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.700688] LNetError: 50808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.720935] LNet: Removed LNI 192.168.206.1@tcp [ 881.747615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.839766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 887.466587] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 08:36:12 (1787574972) [ 888.657414] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 890.216332] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 08:36:15 (1787574975) [ 891.485478] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 893.068922] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 08:36:17 (1787574977) [ 894.438500] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 895.806632] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 08:36:20 (1787574980) [ 897.029299] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 898.559521] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 08:36:23 (1787574983) [ 900.095237] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 902.299968] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 08:36:26 (1787574986) [ 902.446717] LNet: 52855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.247619] Key type .llcrypt unregistered [ 903.250421] Key type ._llcrypt unregistered [ 909.612173] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 909.771963] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 909.776205] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 909.781440] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 912.353399] Key type ._llcrypt registered [ 912.355527] Key type .llcrypt registered [ 912.498523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 912.512588] alg: No test for adler32 (adler32-zlib) [ 913.450571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.578605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 913.607745] LNet: Added LNI 192.168.206.1@tcp [8/512/0/180] [ 913.624505] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 913.630129] LNet: 53187:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 913.759869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.782591] LNet: 53206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.788355] LNetError: 53206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 913.796924] LNet: Removed LNI 10.1.2.3@tcp [ 913.799717] LNetError: 53206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.813644] LNet: Removed LNI 192.168.206.1@tcp [ 913.850393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.955263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 913.977270] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 914.101106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.123150] LNet: 53230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.128927] LNetError: 53230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.144212] LNet: Removed LNI 192.168.206.1@tcp [ 914.166886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.267151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 914.327989] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 914.331980] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 914.338383] LNet: 53238:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 914.428619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.445618] LNet: 53256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.453776] LNetError: 53256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 914.466093] LNet: Removed LNI 10.1.2.3@tcp [ 914.467774] LNetError: 53256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.480068] LNet: Removed LNI 192.168.206.1@tcp [ 914.510979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.584900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 914.602838] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 914.618083] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 914.622861] LNet: 53264:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 914.766543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.788744] LNet: 53290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.793811] LNetError: 53290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 914.798796] LNet: Removed LNI 10.1.2.3@tcp [ 914.801415] LNetError: 53290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.814811] LNet: Removed LNI 192.168.206.1@tcp [ 914.830228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.885631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 914.902709] LNet: Added LNI 192.168.206.1@tcp [8/256/0/182] [ 914.907097] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 915.039529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.070530] LNet: 53318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.077618] LNetError: 53318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 915.082910] LNet: Removed LNI 10.1.2.3@tcp [ 915.088507] LNetError: 53318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.097874] LNet: Removed LNI 192.168.206.1@tcp [ 915.116512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.205782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 915.240793] LNet: Added LNI 192.168.206.1@tcp [8/256/0/182] [ 915.257856] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 915.264471] LNet: 53326:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 915.359840] LNet: Removed LNI 10.1.2.3@tcp [ 915.379759] LNet: Removed LNI 192.168.206.1@tcp [ 916.005880] Key type .llcrypt unregistered [ 916.008503] Key type ._llcrypt unregistered [ 927.230849] Key type ._llcrypt registered [ 927.232451] Key type .llcrypt registered [ 928.591759] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 08:36:53 (1787575013) [ 930.211594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 932.327972] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 08:36:56 (1787575016) [ 932.812535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 932.841801] alg: No test for adler32 (adler32-zlib) [ 933.816721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.911414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 933.946929] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 934.885418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.1 [ 934.905242] LNet: 54594:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 934.912743] LNet: Removed LNI 192.168.206.1@tcp [ 942.309920] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 08:37:06 (1787575026) [ 942.439650] LNet: 55157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.448255] LNetError: 55157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.467609] LNet: Removed LNI 192.168.206.1@tcp [ 943.236589] Key type .llcrypt unregistered [ 943.247768] Key type ._llcrypt unregistered [ 949.722466] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 949.978947] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 952.697256] Key type ._llcrypt registered [ 952.699254] Key type .llcrypt registered [ 952.902104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 952.932983] alg: No test for adler32 (adler32-zlib) [ 954.337083] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 960.610062] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 08:37:25 (1787575045) [ 960.709940] LNet: 56073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.725296] LNetError: 56073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 960.739931] LNet: Removed LNI 10.1.2.3@tcp [ 961.427259] Key type .llcrypt unregistered [ 961.431849] Key type ._llcrypt unregistered [ 967.584041] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 967.807069] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 967.811977] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 967.821915] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 970.474201] Key type ._llcrypt registered [ 970.476442] Key type .llcrypt registered [ 970.655288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 970.671330] alg: No test for adler32 (adler32-zlib) [ 971.605773] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 977.743442] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 08:37:42 (1787575062) [ 979.140307] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 980.627865] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 08:37:45 (1787575065) [ 980.721911] LNet: 57261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.735291] LNetError: 57261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 980.751681] LNet: Removed LNI 10.1.2.3@tcp [ 981.441660] Key type .llcrypt unregistered [ 981.446251] Key type ._llcrypt unregistered [ 987.934842] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 988.156910] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 988.167171] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 988.176067] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 990.897356] Key type ._llcrypt registered [ 990.899316] Key type .llcrypt registered [ 991.086728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 991.114485] alg: No test for adler32 (adler32-zlib) [ 992.068190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.138726] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1001.677234] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 08:38:06 (1787575086) [ 1002.214437] LNet: 58503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.223710] LNetError: 58503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1002.236815] LNet: Removed LNI 10.1.2.3@tcp [ 1003.072730] Key type .llcrypt unregistered [ 1003.075195] Key type ._llcrypt unregistered [ 1009.292105] Key type ._llcrypt registered [ 1009.302296] Key type .llcrypt registered [ 1009.647935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.681458] alg: No test for adler32 (adler32-zlib) [ 1010.666328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1010.689768] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1018.360525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1018.987637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_interrupt -r 1 [ 1019.030511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_interrupt -r 1 [ 1019.075994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1019.100263] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1019.325819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1019.356300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1019.378067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1019.400832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1019.422302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1019.452637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1019.474898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1019.497433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1019.510793] LNet: 58944:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1019.523846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1019.548963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1019.572414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1019.600201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1019.633250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1019.662300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.690499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1020.150282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_dropped -r 1 [ 1020.186240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_dropped -r 1 [ 1020.218877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1020.241509] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1020.436659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1020.473578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1020.512452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1020.554238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1020.588854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1020.616520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1020.644324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1020.669269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1020.686299] LNet: 59054:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1020.704244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1020.735925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1020.764199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1020.789404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1020.851578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1020.895842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1020.932891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1021.468217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_aborted -r 1 [ 1021.520979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_aborted -r 1 [ 1021.581108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1021.599989] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1021.848130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1021.883668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1021.942616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1021.969672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1021.986464] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 1021.998871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1022.027786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1022.069853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1022.107372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1022.125790] LNet: 59163:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1022.144322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1022.170700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1022.194534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1022.221285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1022.251558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1022.292147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1022.323490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1022.829974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_no_route -r 1 [ 1022.881446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_no_route -r 1 [ 1022.934427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1023.227691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1023.261844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1023.286470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1023.330813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1023.385802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.423781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1023.469982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.504994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.535762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1023.563536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1023.588100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1023.610727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1023.634863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1023.669503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1023.712917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1024.182151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_timeout -r 1 [ 1024.218476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_timeout -r 1 [ 1024.258977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1024.279520] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1024.299596] LNet: Skipped 2 previous similar messages [ 1024.533211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1024.551850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1024.571725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1024.602177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1024.642699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1024.689483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1024.732468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1024.777441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1024.804949] LNet: 59381:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1024.815563] LNet: 59381:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1024.832871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1024.863043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1024.907828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1024.935656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1024.985410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1025.044822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1025.078349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1025.684714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_error -r 1 [ 1025.731762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_error -r 1 [ 1025.792811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1026.044552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1026.074928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1026.115157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1026.134665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1026.174982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1026.212545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1026.254902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1026.291952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1026.346966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1026.406424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1026.450255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1026.503422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1026.545715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1026.584941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1026.632105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1027.015749] LNet: 59535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.026371] LNetError: 59535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.042717] LNet: Removed LNI 192.168.206.1@tcp [ 1027.370715] Key type .llcrypt unregistered [ 1027.374231] Key type ._llcrypt unregistered [ 1045.937988] Key type ._llcrypt registered [ 1045.941796] Key type .llcrypt registered [ 1047.620471] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 08:38:51 (1787575131) [ 1048.519448] Key type .llcrypt unregistered [ 1048.522282] Key type ._llcrypt unregistered [ 1055.836866] Key type ._llcrypt registered [ 1055.839019] Key type .llcrypt registered [ 1056.062659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1056.080611] alg: No test for adler32 (adler32-zlib) [ 1057.003409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1057.041789] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1064.751665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1074.698497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1074.730512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1074.775365] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1076.249415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_interrupt -r 1 [ 1076.291769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_interrupt -r 1 [ 1076.330305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_interrupt -r 1 [ 1076.365340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_interrupt -r 1 [ 1076.412970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_interrupt -r 1 [ 1076.487463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_interrupt -r 1 [ 1076.549384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_interrupt -r 1 [ 1076.585251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_interrupt -r 1 [ 1076.637606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1076.672740] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1076.680537] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 1076.921663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1076.953208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1076.994932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1077.025632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1077.061155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1077.126548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1077.179937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1077.241840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1077.266783] LNet: 61008:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1077.289725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1077.327380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1077.368655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1077.402814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1077.450153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1077.491898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1077.521280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1078.012260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_dropped -r 1 [ 1078.071122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_dropped -r 1 [ 1078.116758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_dropped -r 1 [ 1078.155819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_dropped -r 1 [ 1078.195278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_dropped -r 1 [ 1078.247398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_dropped -r 1 [ 1078.293496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_dropped -r 1 [ 1078.354879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_dropped -r 1 [ 1078.395191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1078.423027] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1078.428713] LNet: Skipped 2 previous similar messages [ 1078.672876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1078.725624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1078.766912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1078.809936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1078.841508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1078.893331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1078.931548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1078.963482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1078.982674] LNet: 61129:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1079.000535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1079.045699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1079.102786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1079.133611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1079.164150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1079.208225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1079.242982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1079.768022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_aborted -r 1 [ 1079.818864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_aborted -r 1 [ 1079.890423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_aborted -r 1 [ 1079.952374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_aborted -r 1 [ 1079.994838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_aborted -r 1 [ 1080.048770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_aborted -r 1 [ 1080.086541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_aborted -r 1 [ 1080.125214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_aborted -r 1 [ 1080.152143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1080.177688] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1080.186137] LNet: Skipped 2 previous similar messages [ 1080.400596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1080.443482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1080.496752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1080.545762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1080.582586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1080.632703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1080.681498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1080.713181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1080.729441] LNet: 61250:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1080.743430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1080.766191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1080.791093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1080.819769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1080.842834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1080.869810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1080.896271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1081.347686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_no_route -r 1 [ 1081.391972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_no_route -r 1 [ 1081.447989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_no_route -r 1 [ 1081.486557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_no_route -r 1 [ 1081.537209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_no_route -r 1 [ 1081.573559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_no_route -r 1 [ 1081.597975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_no_route -r 1 [ 1081.624269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_no_route -r 1 [ 1081.655966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1081.857674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1081.899725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1081.940817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1081.984202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1082.019715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1082.058475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1082.089817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1082.129162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1082.159942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1082.202904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1082.247760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1082.284992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1082.319491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1082.352315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1082.390366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1082.879672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_timeout -r 1 [ 1082.903592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_timeout -r 1 [ 1082.931503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_timeout -r 1 [ 1082.957416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_timeout -r 1 [ 1082.986795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_timeout -r 1 [ 1083.014542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_timeout -r 1 [ 1083.041344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_timeout -r 1 [ 1083.066610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_timeout -r 1 [ 1083.095350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1083.110254] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1083.114698] LNet: Skipped 5 previous similar messages [ 1083.313747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1083.341931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1083.364287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1083.385915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1083.407609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1083.429293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1083.466924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1083.511773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1083.548082] LNet: 61493:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1083.559300] LNet: 61493:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1083.581846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1083.640661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1083.715112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1083.765347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1083.790977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1083.846170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1083.888267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1084.397278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_error -r 1 [ 1084.436662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_error -r 1 [ 1084.470807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_error -r 1 [ 1084.498817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_error -r 1 [ 1084.524407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_error -r 1 [ 1084.555697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_error -r 1 [ 1084.579745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_error -r 1 [ 1084.604590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_error -r 1 [ 1084.634152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1084.900573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1084.932547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1084.961612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1085.002957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1085.036770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1085.069902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1085.093383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1085.122425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1085.146797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1085.170798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1085.198860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1085.221673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1085.255867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1085.307730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1085.359133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1090.192918] LNet: 61694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1090.199304] LNetError: 61694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1090.206395] LNet: Removed LNI 192.168.206.1@tcp [ 1091.246062] LNet: Removed LNI 192.168.206.1@tcp1 [ 1091.710567] Key type .llcrypt unregistered [ 1091.713680] Key type ._llcrypt unregistered [ 1107.602394] Key type ._llcrypt registered [ 1107.605854] Key type .llcrypt registered [ 1109.024856] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 08:39:53 (1787575193) [ 1110.008309] Key type .llcrypt unregistered [ 1110.011992] Key type ._llcrypt unregistered [ 1115.601832] Key type ._llcrypt registered [ 1115.604897] Key type .llcrypt registered [ 1115.776761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1115.788611] alg: No test for adler32 (adler32-zlib) [ 1116.721413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1116.742448] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1123.312955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1124.150850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e remote_dropped -r 1 [ 1124.175586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e remote_dropped -r 1 [ 1124.206125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1124.219563] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1124.420422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.444620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1124.467554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.487768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.511408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.535860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.558081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.583856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.596688] LNet: 62956:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.611669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.634671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.656934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.680569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.707946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.733410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.755659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1125.170669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e remote_error -r 1 [ 1125.205619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e remote_error -r 1 [ 1125.250594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1125.269511] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1125.457318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1125.480985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1125.505487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1125.535839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1125.563731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1125.590493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1125.628251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1125.655272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1125.666969] LNet: 63065:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1125.680988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1125.712986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1125.734716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1125.770436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1125.813389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1125.858501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1125.910539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1126.373723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e remote_timeout -r 1 [ 1126.406976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e remote_timeout -r 1 [ 1126.437543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1126.459962] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1126.654934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.683747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1126.716980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.743287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.774594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.810176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.839492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.879599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1126.895524] LNet: 63174:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1126.911312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1126.937298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1126.977835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1127.018544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.067588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1127.114701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1127.161469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1127.429393] LNet: 63219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.442443] LNetError: 63219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.483992] LNet: Removed LNI 192.168.206.1@tcp [ 1127.830765] Key type .llcrypt unregistered [ 1127.833970] Key type ._llcrypt unregistered [ 1142.124939] Key type ._llcrypt registered [ 1142.129736] Key type .llcrypt registered [ 1143.610945] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 08:40:28 (1787575228) [ 1144.357353] Key type .llcrypt unregistered [ 1144.360524] Key type ._llcrypt unregistered [ 1149.130553] Key type ._llcrypt registered [ 1149.132529] Key type .llcrypt registered [ 1149.295925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1149.321352] alg: No test for adler32 (adler32-zlib) [ 1150.217694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1150.238109] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1156.772731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1164.426973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1164.452993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1164.494282] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1165.526162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e remote_dropped -r 1 [ 1165.553727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e remote_dropped -r 1 [ 1165.585645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e remote_dropped -r 1 [ 1165.638736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e remote_dropped -r 1 [ 1165.696044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e remote_dropped -r 1 [ 1165.738512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e remote_dropped -r 1 [ 1165.777751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e remote_dropped -r 1 [ 1165.813067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e remote_dropped -r 1 [ 1165.844030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1165.861057] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1165.867462] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.101@tcp [ 1166.066732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.090488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1166.116869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1166.147623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1166.175456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1166.198444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1166.222854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1166.248133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.266443] LNet: 64691:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1166.285106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.320947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.352869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.383853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1166.412897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.440451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.479517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1166.501249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.848907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e remote_error -r 1 [ 1166.879628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e remote_error -r 1 [ 1166.910653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e remote_error -r 1 [ 1166.939416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e remote_error -r 1 [ 1166.973365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e remote_error -r 1 [ 1167.012610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e remote_error -r 1 [ 1167.052477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e remote_error -r 1 [ 1167.085075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e remote_error -r 1 [ 1167.126576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1167.140502] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1167.146694] LNet: Skipped 2 previous similar messages [ 1167.315256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.335290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1167.356800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.381308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.405155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.429952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.453152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.480950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.499413] LNet: 64814:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1167.520385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.551855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.583262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.611973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.656953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.683290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.714239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1167.747149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.168972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e remote_timeout -r 1 [ 1168.197565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e remote_timeout -r 1 [ 1168.242908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e remote_timeout -r 1 [ 1168.285482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e remote_timeout -r 1 [ 1168.319487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e remote_timeout -r 1 [ 1168.365639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e remote_timeout -r 1 [ 1168.398819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e remote_timeout -r 1 [ 1168.423832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e remote_timeout -r 1 [ 1168.454219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1168.469086] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1168.636349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.666205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1168.687323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1168.714863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.737450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.761871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.789769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.824300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.846662] LNet: 64937:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.865397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.897754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.931581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.955734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.980715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.011044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.062176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1169.090507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.501609] LNet: 65019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1173.509676] LNetError: 65019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1174.560419] LNet: Removed LNI 192.168.206.1@tcp [ 1174.562258] LNetError: 65019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1174.581704] LNet: Removed LNI 192.168.206.1@tcp1 [ 1175.003821] Key type .llcrypt unregistered [ 1175.006531] Key type ._llcrypt unregistered [ 1188.990123] Key type ._llcrypt registered [ 1188.992712] Key type .llcrypt registered [ 1190.303583] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 08:41:15 (1787575275) [ 1191.439929] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1192.942795] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 08:41:17 (1787575277) [ 1193.703698] Key type .llcrypt unregistered [ 1193.705589] Key type ._llcrypt unregistered [ 1198.057810] Key type ._llcrypt registered [ 1198.059816] Key type .llcrypt registered [ 1198.229224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1198.242742] alg: No test for adler32 (adler32-zlib) [ 1199.125612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1199.149884] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1204.858230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1205.591484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e network_timeout -r 1 [ 1205.625657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e network_timeout -r 1 [ 1205.659424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1205.682110] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1205.887161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1205.930976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1205.992899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1206.035955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1206.060733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1206.105285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1206.135983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1206.174324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1206.189629] LNet: 66566:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1206.212909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1206.252125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1206.291676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1206.304763] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 1206.306775] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1206.340563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1206.368369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1206.396537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1206.698844] LNet: 66609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1206.707281] LNetError: 66609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.782369] LNet: Removed LNI 192.168.206.1@tcp [ 1208.317601] Key type .llcrypt unregistered [ 1208.320306] Key type ._llcrypt unregistered [ 1227.317623] Key type ._llcrypt registered [ 1227.319510] Key type .llcrypt registered [ 1227.536426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1227.550838] alg: No test for adler32 (adler32-zlib) [ 1228.550819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1228.589846] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1235.213654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1243.651938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1243.688665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1243.721401] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1245.303376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e network_timeout -r 1 [ 1245.332266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e network_timeout -r 1 [ 1245.387785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e network_timeout -r 1 [ 1245.438432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e network_timeout -r 1 [ 1245.480870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e network_timeout -r 1 [ 1245.523580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e network_timeout -r 1 [ 1245.566372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e network_timeout -r 1 [ 1245.618805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e network_timeout -r 1 [ 1245.657844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1245.678547] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1245.857358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1245.890590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1245.912428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1245.933740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1245.958948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1245.983989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1245.989555] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 1246.005649] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.101@tcp [ 1246.021686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1246.048835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1246.063954] LNet: 67492:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1246.093408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1246.122505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1246.159797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1246.188919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1246.210201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1246.254300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1251.074192] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1251.470501] LNet: 67569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.482696] LNetError: 67569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1251.500870] LNet: Removed LNI 192.168.206.1@tcp [ 1251.510937] LNet: Removed LNI 192.168.206.1@tcp1 [ 1251.961673] Key type .llcrypt unregistered [ 1251.965022] Key type ._llcrypt unregistered [ 1267.983926] Key type ._llcrypt registered [ 1267.985799] Key type .llcrypt registered [ 1269.424952] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 08:42:34 (1787575354) [ 1269.845903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1269.870928] alg: No test for adler32 (adler32-zlib) [ 1270.855978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1270.933910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1270.983450] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1271.963809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1271.992493] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1272.942919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 1273.021888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1273.064945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1273.097490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1273.130781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1273.162657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1273.192602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.1@tcp [ 1273.213557] LNetError: 68668:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.1@tcp to recovery [ 1273.952352] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 1273.952406] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1276.001697] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1276.710303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1280.098199] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1281.307828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1281.342941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1281.393919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1281.419520] LNet: 68785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1281.426860] LNetError: 68785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1281.435990] LNet: Removed LNI 192.168.206.1@tcp [ 1281.457142] LNet: Removed LNI 192.168.206.1@tcp1 [ 1281.506676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1281.588467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1281.604517] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1282.180734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1282.197534] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1282.827331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 1282.860551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1282.911623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1282.951502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1282.980805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1283.005849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1283.034993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1283.076298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.1@tcp [ 1283.096789] LNetError: 69058:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.1@tcp to recovery [ 1283.616649] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1286.627676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1289.760355] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1289.766808] LNet: Skipped 1 previous similar message [ 1294.780436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1294.806430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1294.833478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1301.669649] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 08:43:06 (1787575386) [ 1301.964387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1301.993496] LNet: 69775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.999506] LNetError: 69775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1302.007378] LNetError: 69775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1302.013535] LNet: Removed LNI 192.168.206.1@tcp [ 1302.026418] LNet: Removed LNI 192.168.206.1@tcp1 [ 1302.054537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1302.137679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1302.160732] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1302.968467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1303.010584] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1303.972647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 1304.032806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1304.066428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1304.113745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1304.157500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1304.214663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1304.258917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.1@tcp --health 0 [ 1304.361490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1305.184317] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.1@tcp [ 1305.185297] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1305.214235] LNet: Skipped 1 previous similar message [ 1319.790697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1319.951987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1319.985493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1320.020201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.1@tcp --health 500 [ 1320.112843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1326.835808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1326.899499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1326.936924] LNet: 70266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.945574] LNetError: 70266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1326.953528] LNetError: 70266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1326.960949] LNet: Removed LNI 192.168.206.1@tcp [ 1326.974610] LNet: Removed LNI 192.168.206.1@tcp1 [ 1326.996119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1327.035729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1327.067244] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1327.626849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1327.669478] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1328.336056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 1328.380346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1328.441313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1328.475512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1328.506360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1328.534161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1328.562952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1328.587528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.1@tcp --health 0 [ 1329.057684] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1329.062485] LNet: Skipped 3 previous similar messages [ 1329.881979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1339.939705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1339.980462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1340.007307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1346.642638] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 08:43:51 (1787575431) [ 1347.010564] LNet: 71311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1347.018134] LNetError: 71311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1347.023190] LNetError: 71311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1348.064382] LNet: Removed LNI 192.168.206.1@tcp [ 1348.079394] LNet: Removed LNI 192.168.206.1@tcp1 [ 1348.851778] Key type .llcrypt unregistered [ 1348.856300] Key type ._llcrypt unregistered [ 1355.537759] Key type ._llcrypt registered [ 1355.539379] Key type .llcrypt registered [ 1355.724706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1355.736596] alg: No test for adler32 (adler32-zlib) [ 1356.650520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1356.688064] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1366.219509] Lustre: DEBUG MARKER: Initial discovery [ 1366.276809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.101@tcp [ 1368.950397] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1369.043495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1369.106845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.101@tcp [ 1369.160734] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1369.191302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1369.239319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.101@tcp -r 1 -m GET -l 3 [ 1369.308774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.101@tcp -r 1 -m GET -e local_error [ 1369.376575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.101@tcp -r 1 -m PUT -l 6 [ 1370.080337] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 1370.997651] Lustre: DEBUG MARKER: Force oleg601-client.virtnet to discover 192.168.206.101@tcp (in background) [ 1371.079130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.101@tcp [ 1372.823499] Lustre: DEBUG MARKER: Force oleg601-server to discover 192.168.206.1@tcp [ 1374.177217] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1374.981593] Lustre: DEBUG MARKER: Wait for 72071 [ 1376.302663] Lustre: DEBUG MARKER: Finished wait on 72071 [ 1376.349329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1376.382229] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1376.396336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1376.629291] LNet: 72532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1376.634201] LNetError: 72532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1376.654542] LNet: Removed LNI 192.168.206.1@tcp [ 1377.019584] Key type .llcrypt unregistered [ 1377.027887] Key type ._llcrypt unregistered [ 1391.376555] Key type ._llcrypt registered [ 1391.379806] Key type .llcrypt registered [ 1392.642288] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 08:44:37 (1787575477) [ 1393.199277] Key type .llcrypt unregistered [ 1393.203091] Key type ._llcrypt unregistered [ 1398.832987] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1399.033818] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1399.048913] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1399.058576] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1401.583442] Key type ._llcrypt registered [ 1401.587541] Key type .llcrypt registered [ 1401.770550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1401.789876] alg: No test for adler32 (adler32-zlib) [ 1402.686884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1402.738993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1402.754813] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1403.376527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1403.396878] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1410.817717] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 08:44:55 (1787575495) [ 1411.262705] LNet: 74615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.273845] LNetError: 74615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1411.282524] LNet: Removed LNI 10.1.2.3@tcp [ 1411.308130] LNet: Removed LNI 192.168.206.1@tcp [ 1412.002527] Key type .llcrypt unregistered [ 1412.005867] Key type ._llcrypt unregistered [ 1417.871921] Key type ._llcrypt registered [ 1417.877822] Key type .llcrypt registered [ 1418.076937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1418.090162] alg: No test for adler32 (adler32-zlib) [ 1419.005865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1419.066093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1419.097121] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1419.840978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1419.866364] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1420.687412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.1@tcp [ 1421.732803] LNetError: Unexpected error -22 connecting to 192.168.206.1@tcp at host 192.168.206.1:988 [ 1426.177355] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 08:45:10 (1787575510) [ 1426.293272] LNet: 75741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.305292] LNetError: 75741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1426.319528] LNet: Removed LNI 10.1.2.3@tcp [ 1426.338548] LNet: Removed LNI 192.168.206.1@tcp [ 1426.967895] Key type .llcrypt unregistered [ 1426.971922] Key type ._llcrypt unregistered [ 1431.594282] Key type ._llcrypt registered [ 1431.596225] Key type .llcrypt registered [ 1431.784081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1431.810781] alg: No test for adler32 (adler32-zlib) [ 1432.724111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.786727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1432.812646] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1433.716261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1433.754806] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 1434.939635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.1@tcp1 --nid 192.168.206.1@tcp2 [ 1439.678196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp1 192.168.206.1@tcp2 [ 1439.708857] LNetError: 76589:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.1@tcp1 to 192.168.206.1@tcp2 - Route table may be misconfigured [ 1439.746557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp2 192.168.206.1@tcp1 [ 1446.914974] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 08:45:31 (1787575531) [ 1447.013422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1447.059939] LNet: 77155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.065993] LNetError: 77155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1447.075791] LNet: Removed LNI 192.168.206.1@tcp1 [ 1448.109199] LNet: Removed LNI 192.168.206.1@tcp2 [ 1448.153850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1448.222246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1448.254361] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1448.936503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1448.954070] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1449.913183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 1450.044074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp 192.168.206.1@tcp [ 1450.097303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp1 192.168.206.1@tcp1 [ 1450.167440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.1@tcp -d 192.168.206.1@tcp -e network_timeout [ 1450.256575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -e network_timeout [ 1450.299786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -e network_timeout [ 1450.379920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -e network_timeout [ 1450.455847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1450.497639] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1450.527973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1450.650937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1450.831525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1451.296905] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 1457.055305] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 08:45:41 (1787575541) [ 1457.342290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1457.367501] LNet: 78024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.373947] LNetError: 78024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.385533] LNetError: 78024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1457.392940] LNet: Removed LNI 192.168.206.1@tcp [ 1458.407519] LNet: Removed LNI 192.168.206.1@tcp1 [ 1458.476892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1458.643944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1459.060771] LNet: 78065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1459.693485] Key type .llcrypt unregistered [ 1459.697988] Key type ._llcrypt unregistered [ 1471.640726] Key type ._llcrypt registered [ 1471.648615] Key type .llcrypt registered [ 1473.077265] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 08:45:57 (1787575557) [ 1473.533360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1473.569877] alg: No test for adler32 (adler32-zlib) [ 1474.549233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1476.035384] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1478.222788] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 08:46:02 (1787575562) [ 1478.361231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1478.419857] LNet: 79158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1478.452827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.579361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1478.603286] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1479.451969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1479.489439] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1480.502162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1480.554361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp1 [ 1480.596744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp1 [ 1486.984252] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 08:46:11 (1787575571) [ 1488.567892] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg601-server [ 1490.762475] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 08:46:15 (1787575575) [ 1492.866732] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg601-server [ 1494.619764] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 08:46:19 (1787575579) [ 1496.175574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg601-server [ 1497.733882] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 08:46:22 (1787575582) [ 1499.569825] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg601-server [ 1501.375189] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 08:46:26 (1787575586) [ 1503.121270] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg601-server [ 1504.877817] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 08:46:29 (1787575589) [ 1506.624817] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg601-server [ 1508.380339] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 08:46:33 (1787575593) [ 1510.496230] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg601-server [ 1512.950484] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 08:46:37 (1787575597) [ 1515.093477] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg601-server [ 1517.286322] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 08:46:41 (1787575601) [ 1519.985399] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg601-server [ 1522.176720] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 08:46:46 (1787575606) [ 1522.296744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1522.344897] LNet: 83035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1522.356409] LNetError: 83035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1522.365385] LNet: Removed LNI 192.168.206.1@tcp [ 1522.385819] LNet: Removed LNI 192.168.206.1@tcp1 [ 1522.422023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.502297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1522.532695] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1523.429460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1523.521488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1523.734801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1523.788052] LNet: 83189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1523.801458] LNetError: 83189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1523.809978] LNetError: 83189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1523.837707] LNet: Removed LNI 192.168.206.1@tcp [ 1523.908605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1523.981957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1524.003934] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1524.803564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1525.001472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1525.042929] LNetError: 83334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1525.076764] LNet: Removed LNI 192.168.206.1@tcp [ 1525.113428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.162777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1525.195723] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1526.128412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1526.195040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1526.410735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1526.446791] LNet: 83481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.458215] LNet: 83481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1527.532937] LNet: Removed LNI 192.168.206.1@tcp [ 1527.594751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.692266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1527.732914] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1528.607649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1528.839486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.863686] LNetError: 83627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1528.871509] LNetError: 83627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1528.884829] LNet: Removed LNI 192.168.206.1@tcp [ 1528.902404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.962809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1528.992780] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1529.834869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1529.940470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1530.091514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1531.173727] LNet: Removed LNI 192.168.206.1@tcp [ 1531.208510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1531.270769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1531.306430] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1532.241791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1532.552979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1532.567602] LNet: 83919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1532.574580] LNet: 83919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1533.611507] LNet: Removed LNI 192.168.206.1@tcp [ 1533.648149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1533.762155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1533.799850] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1534.564873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1534.650260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1534.830876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.859312] LNetError: 84066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1534.873187] LNetError: 84066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1534.901047] LNet: Removed LNI 192.168.206.1@tcp [ 1534.925396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.028463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1535.058552] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1535.813628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1536.016758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1537.062991] LNet: Removed LNI 192.168.206.1@tcp [ 1537.098513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.176415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1537.202946] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1537.755548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1537.836192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1538.133657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.238936] LNet: Removed LNI 192.168.206.1@tcp [ 1539.260392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.335568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1539.371844] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1540.102088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1540.399141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1541.486521] LNet: Removed LNI 192.168.206.1@tcp [ 1541.535688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.620549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1541.662284] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1542.398837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1542.483155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1542.768338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.806797] LNet: 84651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.811954] LNet: 84651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1543.846853] LNet: Removed LNI 192.168.206.1@tcp [ 1543.882831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.942796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1543.968132] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1544.867368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1545.223294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1545.271075] LNetError: 84797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1545.284495] LNetError: 84797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1545.324838] LNet: Removed LNI 192.168.206.1@tcp [ 1545.370734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.458709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1545.501192] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1546.225984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1546.296415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.342795] LNet: Removed LNI 192.168.206.1@tcp [ 1546.366941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.465414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1546.488086] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1547.268324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1547.354495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1547.426243] LNet: Removed LNI 192.168.206.1@tcp [ 1547.454913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.526963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1547.624778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1547.688711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.763303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1547.846281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1547.879783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.929248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1548.069303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.131793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1548.230612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1548.261075] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1548.982414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1549.026966] LNet: Removed LNI 192.168.206.1@tcp [ 1549.061626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.172237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1555.652986] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 08:47:20 (1787575640) [ 1555.801824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1555.857637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.941488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1555.998888] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/180] [ 1557.368725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.390055] LNet: Removed LNI 192.168.206.1@tcp231 [ 1557.411501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.471987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1557.505061] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/180] [ 1557.514589] LNet: 85964:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1558.150476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1558.199860] LNet: Removed LNI 192.168.206.1@tcp231 [ 1558.220838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1558.275120] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/180] [ 1559.569538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.590768] LNet: 86108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.595819] LNet: 86108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1559.613227] LNet: Removed LNI 192.168.206.1@tcp231 [ 1559.640349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.726625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1559.761684] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/0] [ 1559.766851] LNet: 86116:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1560.589385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1560.683871] LNet: Removed LNI 192.168.206.1@tcp231 [ 1560.714695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1560.771299] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/0] [ 1562.756752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.839232] LNetError: 86259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.857037] LNetError: 86259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1562.885329] LNet: Removed LNI 192.168.206.1@tcp231 [ 1562.952679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.065253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1563.151904] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/60] [ 1563.168744] LNet: 86267:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1564.030712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1564.088526] LNet: Removed LNI 192.168.206.1@tcp231 [ 1564.103496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1564.140621] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/60] [ 1565.620956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.667322] LNet: Removed LNI 192.168.206.1@tcp231 [ 1565.704085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.812825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1565.848231] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/180] [ 1565.857957] LNet: 86419:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1566.280633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1566.313862] LNet: Removed LNI 192.168.206.1@tcp231 [ 1566.330209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1566.353806] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/180] [ 1574.013755] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 08:47:38 (1787575658) [ 1574.224558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.327365] LNet: Removed LNI 192.168.206.1@tcp231 [ 1574.357576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.417028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1574.434896] LNet: Added LNI 192.168.206.1@tcp232 [8/256/0/180] [ 1575.247488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1575.284936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1575.307889] LNet: Removed LNI 192.168.206.1@tcp232 [ 1575.323546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1575.350126] LNet: Added LNI 192.168.206.1@tcp232 [8/256/0/180] [ 1576.197473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1576.229588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1576.266835] LNet: Removed LNI 192.168.206.1@tcp232 [ 1576.286542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1576.311906] LNet: Added LNI 192.168.206.1@tcp232 [8/256/0/180] [ 1576.954477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1577.013963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1577.061881] LNet: Removed LNI 192.168.206.1@tcp232 [ 1577.073773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1577.096993] LNet: Added LNI 192.168.206.1@tcp232 [8/256/0/180] [ 1577.790805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1577.845934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1577.882984] LNet: Removed LNI 192.168.206.1@tcp232 [ 1577.900489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1584.191170] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 08:47:49 (1787575669) [ 1585.247449] Key type .llcrypt unregistered [ 1585.255546] Key type ._llcrypt unregistered [ 1591.218979] Key type ._llcrypt registered [ 1591.222149] Key type .llcrypt registered [ 1591.532966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1591.572399] alg: No test for adler32 (adler32-zlib) [ 1592.783697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1592.816568] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1601.416243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1610.726591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.776707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1610.800337] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1612.434629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_interrupt -r 2 [ 1612.487746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_interrupt -r 2 [ 1612.546329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_interrupt -r 2 [ 1612.591452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_interrupt -r 2 [ 1612.649707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_interrupt -r 2 [ 1612.721573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_interrupt -r 2 [ 1612.790451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_interrupt -r 2 [ 1612.879277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_interrupt -r 2 [ 1612.937793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp 192.168.206.101@tcp [ 1612.973925] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1612.982119] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 1613.264236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1613.314101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1613.376752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1613.416699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1613.485830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1613.535927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1613.611371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1613.661786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1613.688694] LNet: 88923:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1613.705969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1613.738316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1613.765252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1613.790676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1613.818718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1613.856814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1614.524370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_dropped -r 2 [ 1614.573422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_dropped -r 2 [ 1614.629632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_dropped -r 2 [ 1614.675259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_dropped -r 2 [ 1614.720713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_dropped -r 2 [ 1614.763776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_dropped -r 2 [ 1614.803728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_dropped -r 2 [ 1614.848314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_dropped -r 2 [ 1614.895635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp 192.168.206.101@tcp [ 1614.917555] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1615.302722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1615.345898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1615.384484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1615.440478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1615.491992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1615.543743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1615.603454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1615.637830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1615.660142] LNet: 89044:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1615.688527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1615.730769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1615.797423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1615.839440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1615.899648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1615.968372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1616.742429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_aborted -r 2 [ 1616.790800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_aborted -r 2 [ 1616.853901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_aborted -r 2 [ 1616.934942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_aborted -r 2 [ 1617.004262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_aborted -r 2 [ 1617.078535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_aborted -r 2 [ 1617.151594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_aborted -r 2 [ 1617.230881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_aborted -r 2 [ 1617.267528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp 192.168.206.101@tcp [ 1617.312768] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1617.688892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1617.722341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1617.789872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1617.828734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1617.872994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1617.910776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1617.947847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1617.990499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1618.007418] LNet: 89164:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1618.027484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1618.064730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1618.099468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1618.133922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1618.168491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1618.221830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1618.980545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_no_route -r 2 [ 1619.037583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_no_route -r 2 [ 1619.115409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_no_route -r 2 [ 1619.257401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_no_route -r 2 [ 1619.329274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_no_route -r 2 [ 1619.438418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_no_route -r 2 [ 1619.498589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_no_route -r 2 [ 1619.558787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_no_route -r 2 [ 1619.624145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp 192.168.206.101@tcp [ 1619.652634] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1619.966645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1620.028337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1620.070713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1620.106705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1620.162792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1620.242966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1620.301248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1620.337474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1620.358288] LNet: 89285:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1620.377557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1620.416820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1620.463507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1620.516776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1620.556394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1620.605834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1621.166662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_timeout -r 2 [ 1621.216233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_timeout -r 2 [ 1621.253594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_timeout -r 2 [ 1621.298206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_timeout -r 2 [ 1621.329648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_timeout -r 2 [ 1621.362724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_timeout -r 2 [ 1621.400314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_timeout -r 2 [ 1621.444196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_timeout -r 2 [ 1621.501839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp 192.168.206.101@tcp [ 1621.882643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1621.909552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1621.944672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1621.980291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1622.029803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1622.076489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1622.123758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1622.162513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1622.188375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1622.213920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1622.235900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1622.275716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1622.305111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1622.335206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1622.838733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e remote_dropped -r 2 [ 1622.875381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e remote_dropped -r 2 [ 1622.921715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e remote_dropped -r 2 [ 1622.986522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e remote_dropped -r 2 [ 1623.033856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e remote_dropped -r 2 [ 1623.092856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e remote_dropped -r 2 [ 1623.166152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e remote_dropped -r 2 [ 1623.221795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e remote_dropped -r 2 [ 1623.300793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp 192.168.206.101@tcp [ 1623.358397] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.101@tcp [ 1623.604991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1623.631724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1623.658089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1623.682433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1623.706676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1623.733647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1623.756973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1623.780777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1623.803898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1623.829039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1623.851789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1623.875706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1623.901390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1623.928656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1629.150944] LNet: 89603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.155861] LNetError: 89603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1629.167074] LNet: Removed LNI 192.168.206.1@tcp [ 1629.178267] LNet: Removed LNI 192.168.206.1@tcp1 [ 1629.546421] Key type .llcrypt unregistered [ 1629.550076] Key type ._llcrypt unregistered [ 1645.081389] Key type ._llcrypt registered [ 1645.087126] Key type .llcrypt registered [ 1646.692376] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 08:48:51 (1787575731) [ 1647.157773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1647.181863] alg: No test for adler32 (adler32-zlib) [ 1648.139245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.224405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1648.269516] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1648.929899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1648.959987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.1@tcp [ 1648.974264] LNetError: 90581:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.1@tcp to recovery [ 1649.057396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.1@tcp [ 1649.166500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1654.712412] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 08:48:59 (1787575739) [ 1654.812433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.841321] LNet: 91163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.851068] LNetError: 91163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.880768] LNet: Removed LNI 192.168.206.1@tcp [ 1654.898374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.960802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1654.997932] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1655.904079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1655.926076] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1656.962888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.1@tcp [ 1656.980611] LNetError: 91437:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.1@tcp to recovery [ 1656.993375] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 1657.002820] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1657.161842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.1@tcp [ 1663.984493] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 08:49:08 (1787575748) [ 1665.937814] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg601-server [ 1667.886246] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 08:49:12 (1787575752) [ 1669.921710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg601-server [ 1671.565558] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 08:49:16 (1787575756) [ 1673.139962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg601-server [ 1675.154744] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 08:49:19 (1787575759) [ 1675.312534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.335330] LNet: 93045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.353935] LNetError: 93045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1675.358975] LNet: Removed LNI 192.168.206.1@tcp [ 1675.386991] LNet: Removed LNI 192.168.206.1@tcp1 [ 1675.482223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.537255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1675.586192] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1676.522775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1676.589884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1676.625347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1676.711526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1676.750429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1676.813203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1683.155958] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 08:49:28 (1787575768) [ 1683.298121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.330559] LNet: 93775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.343630] LNetError: 93775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.355989] LNetError: 93775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1683.397694] LNet: Removed LNI 192.168.206.1@tcp [ 1683.436966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.514901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1683.540957] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1690.038706] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 08:49:35 (1787575775) [ 1690.123782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.147543] LNet: 94479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.154229] LNetError: 94479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.167754] LNet: Removed LNI 192.168.206.1@tcp [ 1690.212397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.260329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1690.294715] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1697.491636] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 08:49:42 (1787575782) [ 1699.258668] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1700.900195] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 08:49:45 (1787575785) [ 1701.300774] LNet: 95519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.305740] LNetError: 95519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1701.323308] LNet: Removed LNI 192.168.206.1@tcp [ 1701.908657] Key type .llcrypt unregistered [ 1701.910903] Key type ._llcrypt unregistered [ 1708.736116] Key type ._llcrypt registered [ 1708.740550] Key type .llcrypt registered [ 1709.021714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1709.053272] alg: No test for adler32 (adler32-zlib) [ 1710.085754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1710.116764] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1718.242805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1722.782550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.101@tcp [ 1723.174433] LNet: 95945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.182122] LNetError: 95945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.193703] LNet: Removed LNI 192.168.206.1@tcp [ 1723.628766] Key type .llcrypt unregistered [ 1723.636290] Key type ._llcrypt unregistered [ 1735.180386] Key type ._llcrypt registered [ 1735.184319] Key type .llcrypt registered [ 1736.624692] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1738.111452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1739.461107] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 08:50:24 (1787575824) [ 1741.015168] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1742.573904] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 08:50:27 (1787575827) [ 1744.417672] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg601-server [ 1746.006554] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 08:50:30 (1787575830) [ 1747.849492] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg601-server [ 1749.859705] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 08:50:34 (1787575834) [ 1751.048988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1751.148127] alg: No test for adler32 (adler32-zlib) [ 1752.424915] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 1752.741666] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1754.554883] Key type lgssc registered [ 1756.265362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1778.106503] Key type lgssc unregistered [ 1778.437173] LNet: 99124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.448872] LNetError: 99124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.466117] LNet: Removed LNI 192.168.206.1@tcp [ 1779.132703] Key type .llcrypt unregistered [ 1779.134917] Key type ._llcrypt unregistered [ 1792.700240] Key type ._llcrypt registered [ 1792.702307] Key type .llcrypt registered [ 1793.869130] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 08:51:18 (1787575878) [ 1794.235980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1794.248862] alg: No test for adler32 (adler32-zlib) [ 1795.188361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.271541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1795.324934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1795.381613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1795.420904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1802.010388] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 08:51:26 (1787575886) [ 1802.124798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1802.162152] LNet: 100532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1802.202750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.310607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1802.384562] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1803.790182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.832511] LNet: 100684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.840644] LNetError: 100684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.860987] LNet: Removed LNI 192.168.206.1@tcp [ 1803.917170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.012853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1804.043515] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1804.945587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.963216] LNet: 100713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.968745] LNetError: 100713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.001221] LNet: Removed LNI 192.168.206.1@tcp [ 1805.023601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.097809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1805.116059] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1812.146249] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 08:51:36 (1787575896) [ 1812.425458] LNet: 101302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.445265] LNetError: 101302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1812.505628] LNet: Removed LNI 192.168.206.1@tcp [ 1813.392252] Key type .llcrypt unregistered [ 1813.397835] Key type ._llcrypt unregistered [ 1819.409153] Key type ._llcrypt registered [ 1819.410791] Key type .llcrypt registered [ 1819.685656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1819.711549] alg: No test for adler32 (adler32-zlib) [ 1820.666222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.784650] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1820.787912] LNet: *** cfs_fail_loc=e003, val=100*** [ 1821.740909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1821.765150] LNet: *** cfs_fail_loc=e003, val=100*** [ 1821.767664] LNet: Skipped 2 previous similar messages [ 1821.771940] Ping reply received with mapping: [ 1821.773842] found metadata reply NI[192.168.206.1@tcp] [ 1821.778228] 01 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1821.781610] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1821.784824] 5a 5a 5a 5a 5a 5a 5a 5a [ 1822.032665] LNet: 101704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.050629] LNetError: 101704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.076043] LNet: Removed LNI 192.168.206.1@tcp [ 1823.009544] Key type .llcrypt unregistered [ 1823.011084] Key type ._llcrypt unregistered [ 1837.008669] Key type ._llcrypt registered [ 1837.011413] Key type .llcrypt registered [ 1838.878145] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 08:52:02 (1787575922) [ 1839.510192] Key type .llcrypt unregistered [ 1839.519968] Key type ._llcrypt unregistered [ 1847.708429] Key type ._llcrypt registered [ 1847.714020] Key type .llcrypt registered [ 1848.167660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1848.207963] alg: No test for adler32 (adler32-zlib) [ 1849.371381] LNetError: 102762:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1849.820751] Key type .llcrypt unregistered [ 1849.825585] Key type ._llcrypt unregistered [ 1856.055533] Key type ._llcrypt registered [ 1856.063068] Key type .llcrypt registered [ 1857.486981] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 08:52:22 (1787575942) [ 1857.950426] Key type .llcrypt unregistered [ 1857.953556] Key type ._llcrypt unregistered [ 1864.687395] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1864.693956] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1867.160059] Key type ._llcrypt registered [ 1867.162408] Key type .llcrypt registered [ 1867.317313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1867.331801] alg: No test for adler32 (adler32-zlib) [ 1868.213510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.255964] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1876.870672] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 08:52:41 (1787575961) [ 1877.143378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1877.172376] LNet: 104551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.181380] LNetError: 104551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1877.203631] LNet: Removed LNI 10.1.2.3@tcp [ 1877.241762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1877.931488] Key type .llcrypt unregistered [ 1877.936355] Key type ._llcrypt unregistered [ 1884.027862] Key type ._llcrypt registered [ 1884.031598] Key type .llcrypt registered [ 1884.316669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1884.366892] alg: No test for adler32 (adler32-zlib) [ 1885.334918] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1885.395555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1885.575635] LNet: 104843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.580728] LNetError: 104843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1886.630323] LNet: Removed LNI 192.168.206.1@tcp [ 1887.169411] Key type .llcrypt unregistered [ 1887.174612] Key type ._llcrypt unregistered [ 1898.527693] Key type ._llcrypt registered [ 1898.529858] Key type .llcrypt registered [ 1899.661191] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 08:53:04 (1787575984) [ 1900.292754] Key type .llcrypt unregistered [ 1900.300534] Key type ._llcrypt unregistered [ 1906.225258] Key type ._llcrypt registered [ 1906.230162] Key type .llcrypt registered [ 1906.463760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1906.487323] alg: No test for adler32 (adler32-zlib) [ 1913.564099] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 08:53:18 (1787575998) [ 1913.675113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1913.705204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1913.756393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1913.774049] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1914.451757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1914.498214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1914.540403] LNet: Removed LNI 192.168.206.1@tcp [ 1914.769739] LNet: 106599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.214939] Key type .llcrypt unregistered [ 1915.220223] Key type ._llcrypt unregistered [ 1926.535664] Key type ._llcrypt registered [ 1926.537936] Key type .llcrypt registered [ 1927.754559] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 08:53:32 (1787576012) [ 1928.150300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1928.181289] alg: No test for adler32 (adler32-zlib) [ 1929.144936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.184863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1929.207583] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1929.913687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1929.950189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.1@tcp [ 1930.002690] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1930.022451] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1930.033122] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1929, 0, 0 [ 1930.044228] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1930.046028] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1930.052899] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1930.056775] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1929, 0, 0 [ 1930.064977] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1930.068033] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1930.071954] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1930.078202] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1929, 0, 1979 [ 1930.085650] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1930.088817] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1930.097064] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1930.099677] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1929, 0, 1979 [ 1930.106617] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1936.404317] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 08:53:41 (1787576021) [ 1936.765823] LNet: 108138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1936.773432] LNetError: 108138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1936.786277] LNet: Removed LNI 192.168.206.1@tcp [ 1937.349764] Key type .llcrypt unregistered [ 1937.351190] Key type ._llcrypt unregistered [ 1944.077994] Key type ._llcrypt registered [ 1944.079263] Key type .llcrypt registered [ 1944.377777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1944.395844] alg: No test for adler32 (adler32-zlib) [ 1945.455755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1945.488130] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1952.514182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1961.829561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1961.900617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1961.951522] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1963.107240] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1963.279241] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1963.289108] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1963.301719] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1965.671848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1965.703761] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1975.884829] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 08:54:20 (1787576060) [ 1976.177477] LNet: 109646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.183411] LNetError: 109646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1976.193293] LNet: Removed LNI 192.168.206.1@tcp [ 1977.248367] LNet: Removed LNI 192.168.206.1@tcp1 [ 1977.253512] LNetError: 109646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1977.269686] LNetError: 109646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1977.290475] LNet: Removed LNI 10.1.2.3@tcp99 [ 1977.964565] Key type .llcrypt unregistered [ 1977.966879] Key type ._llcrypt unregistered [ 1984.100239] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1984.107474] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1986.645531] Key type ._llcrypt registered [ 1986.651276] Key type .llcrypt registered [ 1986.904230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1986.941932] alg: No test for adler32 (adler32-zlib) [ 1987.828086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.867654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1987.896696] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1988.549871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1988.566557] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1990.600469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1990.646268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1990.683777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1990.702507] LNetError: 110412:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1990.733174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1990.767605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 1991.178124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1991.268452] LNet: 110432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.275738] LNetError: 110432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1991.282121] LNet: Removed LNI 10.1.2.3@tcp [ 1992.293963] LNet: Removed LNI 192.168.206.1@tcp [ 1993.028476] Key type .llcrypt unregistered [ 1993.033417] Key type ._llcrypt unregistered [ 2006.135630] Key type ._llcrypt registered [ 2006.137657] Key type .llcrypt registered [ 2007.609240] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 08:54:52 (1787576092) [ 2007.936589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2007.948633] alg: No test for adler32 (adler32-zlib) [ 2008.839380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2008.885751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2008.903617] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2009.618618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2009.729455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg601-client@tcp [ 2016.142748] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 08:55:01 (1787576101) [ 2016.443626] LNet: 111970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.453166] LNetError: 111970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2016.490332] LNet: Removed LNI 192.168.206.1@tcp [ 2017.029796] Key type .llcrypt unregistered [ 2017.033835] Key type ._llcrypt unregistered [ 2023.723404] Key type ._llcrypt registered [ 2023.731666] Key type .llcrypt registered [ 2024.027481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2024.065530] alg: No test for adler32 (adler32-zlib) [ 2025.035267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2025.075046] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2025.101084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2031.834097] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 08:55:16 (1787576116) [ 2033.420734] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2035.397888] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 08:55:19 (1787576119) [ 2036.960905] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2038.631144] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 08:55:23 (1787576123) [ 2038.744748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2038.793823] LNet: 113403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.798337] LNetError: 113403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.821207] LNet: Removed LNI 192.168.206.1@tcp [ 2038.853623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.950347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2038.978258] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2039.856613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2039.956251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2046.749395] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 08:55:31 (1787576131) [ 2046.893663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2046.928748] LNet: 114117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.937159] LNetError: 114117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.974393] LNet: Removed LNI 192.168.206.1@tcp [ 2047.001462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2047.063210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2047.099299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2047.878544] Key type .llcrypt unregistered [ 2047.879995] Key type ._llcrypt unregistered [ 2060.367689] Key type ._llcrypt registered [ 2060.370752] Key type .llcrypt registered [ 2061.825346] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 08:55:46 (1787576146) [ 2062.235913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2062.252343] alg: No test for adler32 (adler32-zlib) [ 2063.221912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2063.276934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2063.306907] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2064.148601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2064.197561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2070.398684] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 08:55:55 (1787576155) [ 2070.655704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2070.680458] LNet: 115680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.686868] LNetError: 115680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2070.711737] LNet: Removed LNI 192.168.206.1@tcp [ 2070.738726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2070.863581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2070.899212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2079.289185] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 08:56:03 (1787576163) [ 2079.393168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2079.424536] LNet: 116253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.454565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.518849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2080.171385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2080.235136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2080.319839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2081.094899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2081.164847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2081.276079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2081.299721] LNet: 116295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2081.363963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2081.455456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2082.231545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2082.305799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2082.388611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2083.032618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2083.087117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2090.876353] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 08:56:15 (1787576175) [ 2091.010702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2091.062028] LNet: 116889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.129748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2091.276416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2091.299548] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2092.136612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2092.177266] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2093.259717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2093.365468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2093.729965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2093.774468] LNetError: 117174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2093.786542] LNet: Removed LNI 192.168.206.1@tcp [ 2093.809303] LNet: Removed LNI 192.168.206.1@tcp2 [ 2093.859331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2093.940903] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2093.951939] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2093.956431] LNet: 117180:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2094.470877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2094.734519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2094.810832] LNetError: 117217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2094.819319] LNetError: 117217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2094.838600] LNet: Removed LNI 192.168.206.1@tcp [ 2094.877189] LNet: Removed LNI 192.168.206.1@tcp2 [ 2094.901579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2095.001599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2095.059750] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2095.956897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2095.986140] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2097.174950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2097.230161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2097.284942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2097.673963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2097.747431] LNet: 117502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2097.755771] LNet: 117502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2097.767799] LNetError: 117502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2097.771884] LNetError: 117502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2097.784747] LNet: Removed LNI 192.168.206.1@tcp [ 2098.861072] LNet: Removed LNI 192.168.206.1@tcp2 [ 2098.897769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2098.943391] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2098.947757] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2098.951528] LNet: 117508:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2099.260962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2099.470572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2099.511277] LNet: Removed LNI 192.168.206.1@tcp [ 2100.585224] LNet: Removed LNI 192.168.206.1@tcp2 [ 2100.617400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.721560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2100.764225] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2101.693410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2101.728204] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2102.707024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2102.755548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2102.921957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2102.935153] LNetError: 117825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.938845] LNetError: 117825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2103.968587] LNet: Removed LNI 192.168.206.1@tcp [ 2103.980926] LNet: Removed LNI 192.168.206.1@tcp2 [ 2104.050627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2104.190195] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2104.198549] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2104.209535] LNet: 117831:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2104.871088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2105.128707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2106.208719] LNet: Removed LNI 192.168.206.1@tcp [ 2106.224423] LNet: Removed LNI 192.168.206.1@tcp2 [ 2106.280767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.417459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2106.487767] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2107.192335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2107.217481] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2108.587384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2108.654141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2108.757893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2109.217057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2109.288714] LNet: 118152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.300579] LNet: 118152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2110.368982] LNet: Removed LNI 192.168.206.1@tcp [ 2111.412577] LNet: Removed LNI 192.168.206.1@tcp2 [ 2111.486265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2111.608969] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2111.629417] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2111.654576] LNet: 118159:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2112.408786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2112.595510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2112.617825] LNetError: 118196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2112.623916] LNetError: 118196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2112.634976] LNet: Removed LNI 192.168.206.1@tcp [ 2113.702746] LNet: Removed LNI 192.168.206.1@tcp2 [ 2113.732254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.856650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2113.898522] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2115.028915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2115.046665] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2116.226953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2116.272891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2116.342041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2116.768219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2117.856931] LNet: Removed LNI 192.168.206.1@tcp [ 2117.871136] LNet: Removed LNI 192.168.206.1@tcp2 [ 2117.933947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2118.042842] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2118.052154] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2118.065027] LNet: 118499:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2118.788633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2119.198072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2120.288400] LNet: Removed LNI 192.168.206.1@tcp [ 2120.307721] LNet: Removed LNI 192.168.206.1@tcp2 [ 2120.344073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.437738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2120.471037] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2121.326845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2121.344429] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2122.340413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2122.390378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2122.430811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2122.543120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2123.000628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2124.064199] LNet: Removed LNI 192.168.206.1@tcp [ 2125.094894] LNet: Removed LNI 192.168.206.1@tcp2 [ 2125.153618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2125.259914] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2125.265698] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2125.782788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2132.911594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2134.979468] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 08:56:59 (1787576219) [ 2135.188708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2135.236853] LNet: 119619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.242645] LNet: 119619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2135.258676] LNetError: 119619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2135.268868] LNetError: 119619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2135.277912] LNet: Removed LNI 192.168.206.1@tcp [ 2136.311202] LNet: Removed LNI 192.168.206.1@tcp2 [ 2136.366162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2136.765512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2136.796320] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2137.723606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2137.740964] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2138.527894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2138.553324] LNet: Removed LNI 10.1.2.3@tcp [ 2138.575605] LNet: Removed LNI 192.168.206.1@tcp [ 2138.602393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2138.713203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2138.737141] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2138.742135] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2139.031661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2139.055858] LNet: Removed LNI 192.168.206.1@tcp [ 2139.065569] LNet: Removed LNI 10.1.2.3@tcp [ 2146.652450] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 08:57:11 (1787576231) [ 2146.907098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2146.961399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2147.010461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2147.039582] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2147.844268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2147.897071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2147.929518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2147.963947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2147.995856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2148.057190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2148.100450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2148.142409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2148.185747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2148.217098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2148.252352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2148.411424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2154.275354] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 08:57:19 (1787576239) [ 2154.584803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2155.623691] LNet: Removed LNI 192.168.206.1@tcp [ 2155.677100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.788188] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2155.940331] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2155.954917] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2155.963614] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2158.362284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2158.419551] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2159.237040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2159.270944] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2161.463462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2161.524956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2161.621743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.1@tcp -m PUT -e local_timeout -r 1 [ 2161.718273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2161.788693] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2161.791666] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2162.086486] LNetError: 121296:0:(lib-move.c:4801:lnet_parse()) 192.168.206.1@tcp, src 192.168.206.1@tcp: Dropping PUT (error -108 looking up sender) [ 2163.104862] LNet: Removed LNI 10.1.2.3@tcp [ 2164.138172] LNet: Removed LNI 192.168.206.1@tcp [ 2164.582677] Key type .llcrypt unregistered [ 2164.588958] Key type ._llcrypt unregistered [ 2178.661530] Key type ._llcrypt registered [ 2178.664028] Key type .llcrypt registered [ 2180.263434] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 08:57:44 (1787576264) [ 2180.811480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2180.834676] alg: No test for adler32 (adler32-zlib) [ 2181.818223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.298761] LNet: 122612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2183.382276] Key type .llcrypt unregistered [ 2183.387273] Key type ._llcrypt unregistered [ 2191.379687] Key type ._llcrypt registered [ 2191.394613] Key type .llcrypt registered [ 2191.707143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2191.746181] alg: No test for adler32 (adler32-zlib) [ 2192.749838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2192.786337] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2202.266812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 2202.503568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2203.349846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.101@tcp [ 2203.752312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2203.837735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2203.891905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2203.955093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2204.018497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2204.067247] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.101@tcp [ 2204.125765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2204.177378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2204.239738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2204.269953] LNet: 123066:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2204.320567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2204.390278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2204.424613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2204.456488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2204.489162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2204.565639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2205.432927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2205.836202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2205.887308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2205.927945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2205.987647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2206.048901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2206.104758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2206.166868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2206.209651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2206.239701] LNet: 123183:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2206.269792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2206.315801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2206.384341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2206.421223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2206.484768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2206.522284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2207.346522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.101@tcp [ 2207.577870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2207.605742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2207.639378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2207.675412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2207.703609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2207.732509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2207.774226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2207.811571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2207.846698] LNet: 123301:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2207.874621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2207.927928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2207.986917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2208.044449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2208.091881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2208.144211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2209.049887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2209.462635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2209.510810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2209.567375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2209.624192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2209.701304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2209.758272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2209.830110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2209.883086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2209.906762] LNet: 123418:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2209.921427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2209.964796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2210.044926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2210.105833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2210.161659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2210.211624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2210.641061] LNet: 123468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2210.650223] LNetError: 123468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2210.669809] LNet: Removed LNI 192.168.206.1@tcp [ 2210.973368] Key type .llcrypt unregistered [ 2210.977238] Key type ._llcrypt unregistered [ 2229.154484] Key type ._llcrypt registered [ 2229.158802] Key type .llcrypt registered [ 2230.776188] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 08:58:35 (1787576315) [ 2231.414519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2231.430514] alg: No test for adler32 (adler32-zlib) [ 2232.446703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2233.000774] LNet: 124355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2233.810344] Key type .llcrypt unregistered [ 2233.815108] Key type ._llcrypt unregistered [ 2241.732515] Key type ._llcrypt registered [ 2241.734806] Key type .llcrypt registered [ 2241.958448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2241.981192] alg: No test for adler32 (adler32-zlib) [ 2242.960322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2242.997336] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2251.236634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 2261.012915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.084484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2261.097919] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 2261.875653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2262.427503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.101@tcp [ 2262.504124] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.101@tcp [ 2262.725477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2262.754887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2262.781811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2262.811691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2262.859136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2262.897347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2262.941147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2262.989607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2263.022503] LNet: 125012:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2263.048879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2263.094317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2263.182556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2263.224639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2263.268920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2263.303996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2264.228589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2264.536289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2264.600194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2264.628443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2264.671964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2264.730842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2264.750571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2264.783543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2264.873717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2264.899979] LNet: 125129:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2264.926620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2264.976918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2265.002830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2265.027934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2265.068866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2265.135722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2266.003169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.101@tcp [ 2266.230481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2266.274247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2266.295063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2266.343441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2266.402408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2266.444249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2266.486711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2266.515306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2266.532583] LNet: 125246:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2266.546782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2266.666774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2266.719802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2266.786110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2266.855279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2266.944969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2267.805395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2268.102125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2268.133251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2268.158340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2268.182635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2268.207122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2268.229388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2268.260197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2268.289527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2268.321805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2268.346838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2268.386983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2268.426208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2268.445255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2268.515478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2274.865164] LNet: 125448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2274.871538] LNetError: 125448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2275.936649] LNet: Removed LNI 192.168.206.1@tcp [ 2275.940393] LNetError: 125448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2275.961974] LNet: Removed LNI 192.168.206.1@tcp1 [ 2276.464352] Key type .llcrypt unregistered [ 2276.466893] Key type ._llcrypt unregistered [ 2293.543081] Key type ._llcrypt registered [ 2293.547125] Key type .llcrypt registered [ 2294.746875] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 08:59:39 (1787576379) [ 2295.389729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2295.403509] alg: No test for adler32 (adler32-zlib) [ 2296.373017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2296.450966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2296.486821] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2297.373129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2297.388426] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2298.315938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2298.412814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2299.489055] LNet: Removed LNI 192.168.206.1@tcp [ 2305.713471] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 08:59:50 (1787576390) [ 2307.001780] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2308.998694] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 08:59:53 (1787576393) [ 2316.433574] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 09:00:00 (1787576400) [ 2324.915749] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 09:00:09 (1787576409) [ 2332.099753] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 09:00:17 (1787576417) [ 2339.827365] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 09:00:24 (1787576424) [ 2347.637861] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 09:00:32 (1787576432) [ 2354.976351] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 09:00:39 (1787576439) [ 2363.554163] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 09:00:48 (1787576448) [ 2370.534457] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 09:00:55 (1787576455) [ 2376.381334] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 09:01:01 (1787576461) [ 2383.438765] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 09:01:08 (1787576468) [ 2390.664892] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 09:01:15 (1787576475) [ 2397.587224] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 09:01:22 (1787576482) [ 2405.719138] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 09:01:29 (1787576489) [ 2413.741832] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 09:01:38 (1787576498) [ 2423.036945] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 09:01:47 (1787576507) [ 2432.067263] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 09:01:56 (1787576516) [ 2440.458591] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 09:02:05 (1787576525) [ 2440.868807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2440.906661] LNet: 137157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2440.914719] LNetError: 137157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2440.932896] LNet: Removed LNI 192.168.206.1@tcp2 [ 2440.971711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2441.174425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2441.214436] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2442.580356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2442.610356] LNet: 137299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2442.615844] LNetError: 137299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2442.634752] LNet: Removed LNI 192.168.206.1@tcp [ 2442.660158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2442.724324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2442.747051] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2450.253453] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 09:02:14 (1787576534) [ 2457.524292] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 09:02:22 (1787576542) [ 2465.147862] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 09:02:30 (1787576550) [ 2472.819959] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 09:02:37 (1787576557) [ 2478.978393] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 09:02:43 (1787576563) [ 2486.775655] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 09:02:51 (1787576571) [ 2494.727387] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 09:02:59 (1787576579) [ 2503.975402] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 09:03:08 (1787576588) [ 2512.419384] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 09:03:17 (1787576597) [ 2521.968279] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 09:03:26 (1787576606) [ 2531.222420] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 09:03:35 (1787576615) [ 2540.160164] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 09:03:44 (1787576624) [ 2548.492593] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2322 sec ========== 09:03:53 (1787576633) [ 2548.829299] LNet: 144768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2548.841787] LNetError: 144768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.871984] LNet: Removed LNI 192.168.206.1@tcp [ 2549.588724] Key type .llcrypt unregistered [ 2549.592806] Key type ._llcrypt unregistered [ 2561.565232] Key type ._llcrypt registered [ 2561.567811] Key type .llcrypt registered [ 2561.883248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2561.915256] alg: No test for adler32 (adler32-zlib) [ 2563.369810] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 2563.893332] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2565.728179] Key type lgssc registered [ 2567.435514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2637.464583] Lustre: Mounted lustre-client - version 2.17.57_80_g1214e44 [ 2642.880813] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2652.244543] Lustre: Unmounted lustre-client [ 2674.203884] Key type lgssc unregistered [ 2674.559614] LNet: 147727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2674.576301] LNetError: 147727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2674.601423] LNet: Removed LNI 192.168.206.1@tcp [ 2675.848221] Key type .llcrypt unregistered [ 2675.868600] Key type ._llcrypt unregistered