[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 322041863 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002144] x2apic enabled [ 0.002820] Switched APIC routing to physical x2apic. [ 0.003005] kvm-guest: setup PV IPIs [ 0.005512] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006008] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007004] pid_max: default: 32768 minimum: 301 [ 0.007913] LSM: Security Framework initializing [ 0.008022] Yama: becoming mindful. [ 0.009016] SELinux: Initializing. [ 0.010031] *** VALIDATE selinux *** [ 0.015518] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.018515] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.019070] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.020052] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.021045] *** VALIDATE tmpfs *** [ 0.022307] *** VALIDATE proc *** [ 0.023122] *** VALIDATE cgroup *** [ 0.024003] *** VALIDATE cgroup2 *** [ 0.025044] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.026084] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.027002] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.028018] Spectre V2 : User space: Vulnerable [ 0.029005] Speculative Store Bypass: Vulnerable [ 0.032132] debug: unmapping init [mem 0xffffffff89059000-0xffffffff89060fff] [ 0.034106] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.035404] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.036008] ... version: 2 [ 0.036726] ... bit width: 48 [ 0.037005] ... generic registers: 4 [ 0.038005] ... value mask: 0000ffffffffffff [ 0.038942] ... max period: 00007fffffffffff [ 0.039005] ... fixed-purpose events: 3 [ 0.039832] ... event mask: 000000070000000f [ 0.040140] rcu: Hierarchical SRCU implementation. [ 0.041992] smp: Bringing up secondary CPUs ... [ 0.042473] x86: Booting SMP configuration: [ 0.043010] .... node #0, CPUs: #1 #2 #3 [ 0.045180] smp: Brought up 1 node, 4 CPUs [ 0.046979] smpboot: Max logical packages: 1 [ 0.047006] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.127417] node 0 deferred pages initialised in 79ms [ 0.129279] devtmpfs: initialized [ 0.130149] x86/mm: Memory block size: 128MB [ 0.131940] gcov: version magic: 0x41383552 [ 0.132458] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.133042] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.134173] pinctrl core: initialized pinctrl subsystem [ 0.135146] [ 0.135709] ************************************************************* [ 0.136007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.137007] ** ** [ 0.138006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.139007] ** ** [ 0.140007] ** This means that this kernel is built to expose internal ** [ 0.141007] ** IOMMU data structures, which may compromise security on ** [ 0.142011] ** your system. ** [ 0.143014] ** ** [ 0.144010] ** If you see this message and you are not debugging the ** [ 0.145011] ** kernel, report this immediately to your vendor! ** [ 0.146007] ** ** [ 0.147008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148007] ************************************************************* [ 0.149557] NET: Registered protocol family 16 [ 0.150405] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.151034] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.152034] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.153390] cpuidle: using governor menu [ 0.154000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.156300] PCI: Using configuration type 1 for base access [ 0.157073] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.163101] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.164010] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.165086] cryptd: max_cpu_qlen set to 1000 [ 0.166134] ACPI: Added _OSI(Module Device) [ 0.167008] ACPI: Added _OSI(Processor Device) [ 0.167768] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.169008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.173091] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.177352] ACPI: Interpreter enabled [ 0.178040] ACPI: PM: (supports S0 S3 S4 S5) [ 0.178804] ACPI: Using IOAPIC for interrupt routing [ 0.179042] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.181221] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.187678] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.189014] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.190007] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.192054] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.194881] acpiphp: Slot [2] registered [ 0.196060] acpiphp: Slot [5] registered [ 0.197010] acpiphp: Slot [6] registered [ 0.197938] acpiphp: Slot [3] registered [ 0.199049] acpiphp: Slot [4] registered [ 0.199732] acpiphp: Slot [7] registered [ 0.200039] acpiphp: Slot [8] registered [ 0.200866] acpiphp: Slot [9] registered [ 0.202040] acpiphp: Slot [10] registered [ 0.202904] acpiphp: Slot [11] registered [ 0.204051] acpiphp: Slot [12] registered [ 0.204900] acpiphp: Slot [13] registered [ 0.206062] acpiphp: Slot [14] registered [ 0.207053] acpiphp: Slot [15] registered [ 0.208050] acpiphp: Slot [16] registered [ 0.208920] acpiphp: Slot [17] registered [ 0.210053] acpiphp: Slot [18] registered [ 0.210901] acpiphp: Slot [19] registered [ 0.212043] acpiphp: Slot [20] registered [ 0.213040] acpiphp: Slot [21] registered [ 0.213911] acpiphp: Slot [22] registered [ 0.214041] acpiphp: Slot [23] registered [ 0.214873] acpiphp: Slot [24] registered [ 0.216050] acpiphp: Slot [25] registered [ 0.216884] acpiphp: Slot [26] registered [ 0.217041] acpiphp: Slot [27] registered [ 0.217841] acpiphp: Slot [28] registered [ 0.219086] acpiphp: Slot [29] registered [ 0.219841] acpiphp: Slot [30] registered [ 0.221052] acpiphp: Slot [31] registered [ 0.221842] PCI host bridge to bus 0000:00 [ 0.222008] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.224008] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.225007] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.226012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.228018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.230029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.232145] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.233607] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.235835] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.239919] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.243064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.244006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.246010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.248008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.249341] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.250574] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.252020] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.254473] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.255936] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.261007] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.264006] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.267286] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.270817] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.274009] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.286015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.293640] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.299012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.302846] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.313996] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.321886] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.324387] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.326180] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.328266] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.330142] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.333597] iommu: Default domain type: Passthrough [ 0.335440] SCSI subsystem initialized [ 0.337095] ACPI: bus type USB registered [ 0.338064] usbcore: registered new interface driver usbfs [ 0.340049] usbcore: registered new interface driver hub [ 0.342043] usbcore: registered new device driver usb [ 0.343118] pps_core: LinuxPPS API ver. 1 registered [ 0.345008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.348037] PTP clock support registered [ 0.350106] EDAC MC: Ver: 3.0.0 [ 0.352285] PCI: Using ACPI for IRQ routing [ 0.353755] NetLabel: Initializing [ 0.355008] NetLabel: domain hash size = 128 [ 0.356007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.358074] NetLabel: unlabeled traffic allowed by default [ 0.360037] vgaarb: loaded [ 0.362224] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.363007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.367263] clocksource: Switched to clocksource kvm-clock [ 0.447038] VFS: Disk quotas dquot_6.6.0 [ 0.447885] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.449312] *** VALIDATE ramfs *** [ 0.449958] *** VALIDATE hugetlbfs *** [ 0.450736] pnp: PnP ACPI init [ 0.452170] pnp: PnP ACPI: found 6 devices [ 0.464703] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.466426] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.467853] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.469581] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.471400] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.473112] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.475143] NET: Registered protocol family 2 [ 0.476983] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.480634] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.483796] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.488302] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.490696] TCP: Hash tables configured (established 65536 bind 65536) [ 0.492845] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.495034] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.497100] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.499317] NET: Registered protocol family 1 [ 0.501341] RPC: Registered named UNIX socket transport module. [ 0.503494] RPC: Registered udp transport module. [ 0.504986] RPC: Registered tcp transport module. [ 0.506227] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.507852] NET: Registered protocol family 44 [ 0.508939] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.510478] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.512017] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.513775] PCI: CLS 0 bytes, default 64 [ 0.514919] Unpacking initramfs... [ 1.611826] debug: unmapping init [mem 0xffff8b163cc64000-0xffff8b163ffcffff] [ 1.615436] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.617281] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.619578] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.088807] Initialise system trusted keyrings [ 2.090291] Key type blacklist registered [ 2.092733] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.101533] zbud: loaded [ 2.103846] *** VALIDATE nfs *** [ 2.105216] *** VALIDATE nfs4 *** [ 2.106166] pstore: using deflate compression [ 2.108458] Platform Keyring initialized [ 2.194519] NET: Registered protocol family 38 [ 2.195941] Key type asymmetric registered [ 2.197520] Asymmetric key parser 'x509' registered [ 2.199313] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.202321] io scheduler mq-deadline registered [ 2.204126] io scheduler kyber registered [ 2.206236] io scheduler bfq registered [ 2.208339] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.211759] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.214691] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.217556] ACPI: Power Button [PWRF] [ 2.223224] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.230389] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.240604] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.267978] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.295108] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.300514] Non-volatile memory driver v1.3 [ 2.301532] Linux agpgart interface v0.103 [ 2.327728] virtio_blk virtio1: [vda] 146696 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.330912] vda: detected capacity change from 0 to 75108352 [ 2.348846] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.352150] vdb: detected capacity change from 0 to 1073741824 [ 2.358661] libphy: Fixed MDIO Bus: probed [ 2.364489] usbcore: registered new interface driver usbserial_generic [ 2.366244] usbserial: USB Serial support registered for generic [ 2.367783] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.370775] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.372178] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.373968] mousedev: PS/2 mouse device common for all mice [ 2.375986] rtc_cmos 00:05: RTC can wake from S4 [ 2.378878] rtc_cmos 00:05: registered as rtc0 [ 2.380354] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.382297] intel_pstate: CPU model not supported [ 2.385155] hid: raw HID events driver (C) Jiri Kosina [ 2.387318] usbcore: registered new interface driver usbhid [ 2.388022] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.388674] usbhid: USB HID core driver [ 2.393884] drop_monitor: Initializing network drop monitor service [ 2.395624] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.396917] Initializing XFRM netlink socket [ 2.401218] NET: Registered protocol family 10 [ 2.401370] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.403635] Segment Routing with IPv6 [ 2.406968] NET: Registered protocol family 17 [ 2.408420] mpls_gso: MPLS GSO support [ 2.415088] RAS: Correctable Errors collector initialized. [ 2.416950] AVX version of gcm_enc/dec engaged. [ 2.418093] AES CTR mode by8 optimization enabled [ 2.481534] sched_clock: Marking stable (2481429181, 0)->(3079061050, -597631869) [ 2.486541] registered taskstats version 1 [ 2.488502] Loading compiled-in X.509 certificates [ 2.489904] zswap: loaded using pool lzo/zbud [ 2.510700] Key type big_key registered [ 2.521271] Key type encrypted registered [ 2.523231] ima: No TPM chip found, activating TPM-bypass! [ 2.525150] ima: Allocated hash algorithm: sha1 [ 2.527185] ima: No architecture policies found [ 2.528605] evm: Initialising EVM extended attributes: [ 2.530197] evm: security.selinux [ 2.531210] evm: security.ima [ 2.532234] evm: security.capability [ 2.533221] evm: HMAC attrs: 0x1 [ 2.535588] rtc_cmos 00:05: setting system clock to 2026-09-08 01:53:58 UTC (1788832438) [ 2.541742] debug: unmapping init [mem 0xffffffff8a003000-0xffffffff8a1fffff] [ 2.544090] debug: unmapping init [mem 0xffffffff88d82000-0xffffffff89058fff] [ 2.548087] Write protecting the kernel read-only data: 28672k [ 2.550970] debug: unmapping init [mem 0xffffffff87403000-0xffffffff875fffff] [ 2.553096] debug: unmapping init [mem 0xffffffff87d14000-0xffffffff87dfffff] [ 2.579929] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.587253] systemd[1]: Detected virtualization kvm. [ 2.588824] systemd[1]: Detected architecture x86-64. [ 2.590617] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.617327] systemd[1]: No hostname configured. [ 2.618583] systemd[1]: Set hostname to . [ 2.620086] random: systemd: uninitialized urandom read (16 bytes read) [ 2.621943] systemd[1]: Initializing machine ID from random generator. [ 2.742453] random: systemd: uninitialized urandom read (16 bytes read) [ 2.744851] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 2.748526] random: systemd: uninitialized urandom read (16 bytes read) [ 2.750738] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 2.754691] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.473102] device-mapper: uevent: version 1.0.3 [ 3.475371] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.294636] virtio_net virtio0 ens2: renamed from eth0 [ 4.297286] scsi host0: ata_piix [ 4.373511] scsi host1: ata_piix [ 4.379088] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.383796] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.048378] random: crng init done [ 9.053858] random: 7 urandom warning(s) missed due to ratelimiting [ 9.118638] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.379572] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.880739] printk: systemd: 23 output lines suppressed due to ratelimiting [ 13.426589] SELinux: Disabled at runtime. [ 13.511241] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.524614] systemd[1]: Detected virtualization kvm. [ 13.525971] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.786639] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.792470] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.802842] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.818623] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.829937] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.845963] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.865677] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ 15.000660] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 15.900766] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.490865] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.641234] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.789786] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.806513] EDAC sbridge: Ver: 1.1.2 [ 18.709679] Key type dns_resolver registered [ 19.259843] NFS: Registering the id_resolver key type [ 19.262181] Key type id_resolver registered [ 19.264393] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg647-client login: [ 67.050170] libcfs: loading out-of-tree module taints kernel. [ 67.482113] Key type ._llcrypt registered [ 67.488792] Key type .llcrypt registered [ 68.771387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 68.791854] alg: No test for adler32 (adler32-zlib) [ 70.838795] hrtimer: interrupt took 43785885 ns [ 71.859406] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 74.550143] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 76.671226] Key type lgssc registered [ 82.671799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 248.575344] Lustre: Mounted lustre-client - version 2.17.57_110_g063755c [ 253.461307] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 266.045354] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing check_logdir /tmp/testlogs/ [ 271.191451] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing yml_node [ 274.400492] Lustre: lustre-OST0000-osc-ffff8b1698267000: disconnect after 24s idle [ 276.178811] Lustre: DEBUG MARKER: Client: 2.17.57.110 [ 278.852471] Lustre: DEBUG MARKER: MDS: 2.17.57.110 [ 281.104381] Lustre: DEBUG MARKER: OSS: 2.17.57.110 [ 282.399533] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Sep 7 21:58:37 EDT 2026 [ 297.380863] Lustre: Unmounted lustre-client [ 312.464580] Key type lgssc unregistered [ 312.691428] LNet: 8416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.700725] LNetError: 8416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 312.721066] LNet: Removed LNI 192.168.206.47@tcp [ 313.392979] Key type .llcrypt unregistered [ 313.394561] Key type ._llcrypt unregistered [ 338.006298] Key type ._llcrypt registered [ 338.009493] Key type .llcrypt registered [ 338.160972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 338.177718] alg: No test for adler32 (adler32-zlib) [ 339.067721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 339.131927] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 339.152310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 342.913627] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 343.108121] LNet: 9294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.121661] LNetError: 9294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 343.146150] LNet: Removed LNI 192.168.206.47@tcp [ 343.643263] Key type .llcrypt unregistered [ 343.645118] Key type ._llcrypt unregistered [ 349.006729] Key type ._llcrypt registered [ 349.009657] Key type .llcrypt registered [ 350.133072] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:59:44 (1788832784) [ 350.612099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 350.626192] alg: No test for adler32 (adler32-zlib) [ 351.594948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.712159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.731422] LNet: 9705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.758642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.860776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 351.902700] LNet: 9712:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 359.737376] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:59:54 (1788832794) [ 359.852406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.879807] LNet: 10281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.911937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.991771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 365.880372] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 22:00:00 (1788832800) [ 366.008291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.036858] LNet: 10874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.065490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.131419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 372.288660] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 22:00:06 (1788832806) [ 372.489250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.510727] LNet: 11460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.526460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.589873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 379.412238] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 22:00:14 (1788832814) [ 379.508637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.549203] LNet: 12039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.596686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.721363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 380.168969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 386.680619] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 22:00:21 (1788832821) [ 386.966038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.045309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.149260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 387.639180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 388.303024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 394.575822] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 22:00:29 (1788832829) [ 394.752705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.778363] LNet: 13233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.787593] LNet: 13233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 394.819697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.923725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 401.835591] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 22:00:36 (1788832836) [ 402.052737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.109645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.184415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 402.248703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 402.779178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 402.807531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 403.277338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 403.329421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 403.654839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 403.682440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 404.196372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 404.237264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 404.712974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 404.766460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 405.303889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 405.346942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 405.898351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 405.958980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 406.545583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 406.592269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 407.139420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 407.170248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 407.570106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 407.602097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 407.837141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 407.876615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 408.399685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 408.453503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 414.740415] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 22:00:49 (1788832849) [ 414.852087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.903551] LNet: 14493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.909329] LNet: 14493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 414.946739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.081571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 415.139788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 421.392676] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 22:00:56 (1788832856) [ 421.570378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.600451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.660506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 421.687155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 427.825508] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 22:01:02 (1788832862) [ 428.160764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.245872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.332426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 428.383740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 434.864496] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 22:01:09 (1788832869) [ 435.030832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.121155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.223774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 435.253693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 442.182675] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 22:01:16 (1788832876) [ 442.332174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.412265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.531356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 442.577877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 450.255302] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 22:01:24 (1788832884) [ 450.356715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.410257] LNet: 17438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 450.416881] LNet: 17438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 450.458232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.558257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 450.598658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 457.102457] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 22:01:31 (1788832891) [ 457.190669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.238058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.393859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 457.693917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 465.042410] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 22:01:39 (1788832899) [ 465.180300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.237114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.435562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 465.594368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 471.799487] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 22:01:46 (1788832906) [ 472.092827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.168655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.302336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 472.549576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 479.395744] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 22:01:53 (1788832913) [ 479.493954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.518439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.975202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 480.189886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 487.420795] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 22:02:01 (1788832921) [ 487.566670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.630525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.686230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 487.749554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 493.436336] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 22:02:08 (1788832928) [ 493.638643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.664727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.741437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 493.802221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 501.015237] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 22:02:15 (1788832935) [ 501.161176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.226087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.347449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 508.438387] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 22:02:22 (1788832942) [ 508.552880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.653710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.751381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 516.351364] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 22:02:30 (1788832950) [ 516.558517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.587856] LNet: 22880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 516.595324] LNet: 22880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 516.640467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.750293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 517.064802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 523.538718] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 22:02:38 (1788832958) [ 523.880222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.925670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.003340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 524.035806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 530.309298] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 22:02:44 (1788832964) [ 530.414954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.470513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.542565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 530.577536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 536.141828] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 22:02:50 (1788832970) [ 536.239692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.293208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.374928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 536.403744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 542.259881] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 22:02:56 (1788832976) [ 542.345063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.382603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.480805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 542.529559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 548.078031] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 22:03:02 (1788832982) [ 548.290594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.370023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.432980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 548.539717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 548.567061] LNetError: 25830:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 548.616555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 548.663210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 554.192557] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 22:03:08 (1788832988) [ 554.390364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.442725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.532403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.xjdQLEG45t [ 554.626464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.xjdQLEG45t [ 554.685388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.xjdQLEG45t [ 561.457540] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 22:03:15 (1788832995) [ 561.583064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.625512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.670122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 561.707980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 561.742507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 568.028945] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 22:03:22 (1788833002) [ 568.151866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.220858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.364797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 568.413150] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 570.070770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 571.385092] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 573.199578] Key type lgssc registered [ 574.316704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 577.446285] LNetError: 27589:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.47@tcp99 has no available nets [ 579.834362] Key type lgssc unregistered [ 580.142993] LNetError: 28696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 581.160388] LNet: Removed LNI 192.168.206.47@tcp [ 581.730930] Key type .llcrypt unregistered [ 581.737950] Key type ._llcrypt unregistered [ 587.101687] Key type ._llcrypt registered [ 587.104996] Key type .llcrypt registered [ 588.124346] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 22:03:42 (1788833022) [ 588.599375] Key type .llcrypt unregistered [ 588.605926] Key type ._llcrypt unregistered [ 594.245870] Key type ._llcrypt registered [ 594.248290] Key type .llcrypt registered [ 594.479882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 594.498213] alg: No test for adler32 (adler32-zlib) [ 595.393425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.442768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 595.457403] LNet: 29604:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 595.463166] LNet: 29604:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 595.468872] LNet: 29604:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 595.480947] LNet: Message forwarding will be enabled in 2s [ 598.559127] LNet: Message forwarding enabled [ 598.924953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 598.952267] LNet: Message forwarding will stop in 2s to 6s [ 602.663215] LNet: Message forwarding disabled [ 603.382096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 603.401473] LNet: 29702:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 603.411537] LNet: 29702:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 603.420467] LNet: 29702:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 603.440590] LNet: Message forwarding will be enabled in 2s [ 603.628836] LNet: 29715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 604.234486] Key type .llcrypt unregistered [ 604.243064] Key type ._llcrypt unregistered [ 615.354110] Key type ._llcrypt registered [ 615.356478] Key type .llcrypt registered [ 616.497886] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 22:04:11 (1788833051) [ 617.042200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 617.068269] alg: No test for adler32 (adler32-zlib) [ 617.956644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.028601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 623.445984] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 22:04:18 (1788833058) [ 623.681581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.730553] LNet: 31115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.757470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.826254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 623.869570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 623.923588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 623.973279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 624.025377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 624.070587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 624.109791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 624.154185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 624.204788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 624.248551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 624.303528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 624.345352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 624.384798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 624.441208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 624.489473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 624.543940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 624.586546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 624.621714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 624.657549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 624.680620] LNetError: 31159:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 630.744915] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 22:04:25 (1788833065) [ 630.858384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.908408] LNet: 31724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.946871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.027926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 637.350565] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 22:04:31 (1788833071) [ 639.490198] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg647-server [ 641.534493] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 22:04:35 (1788833075) [ 641.691728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.748552] LNet: 32657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.786228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.859468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 641.878433] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 642.763450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[48-52]@tcp [ 642.796616] LNet: 32793:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 642.860385] LNetError: 32664:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.51@tcp is being used as a gateway but routing feature is not turned on [ 642.880051] LNetError: 32664:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.51@tcp has gone from up to down [ 649.139551] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 22:04:43 (1788833083) [ 650.949544] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg647-server [ 653.086663] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 22:04:47 (1788833087) [ 653.203274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.231316] LNet: 33722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.240646] LNetError: 33722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 653.269800] LNet: Removed LNI 192.168.206.47@tcp [ 653.288897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.381227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 653.424866] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 654.063866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[48-53/2]@tcp [ 654.080736] LNet: 33846:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 654.088570] LNet: 33846:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 654.107283] LNetError: 33733:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.50@tcp is being used as a gateway but routing feature is not turned on [ 654.120054] LNetError: 33733:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 654.128022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[48-53/2]@tcp [ 654.135410] LNetError: 33733:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.50@tcp has gone from up to down [ 654.135420] LNetError: 33733:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 660.847443] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 22:04:55 (1788833095) [ 661.015204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.064181] LNet: 34432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.076180] LNetError: 34432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 661.106082] LNet: Removed LNI 192.168.206.47@tcp [ 661.178685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.319507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 661.393735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.434152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.503647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 661.546377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.600351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.645818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 661.734220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.789630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.893608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 661.987768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.027863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.113812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 662.228924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.298402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.412627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 662.518715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.609399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.694813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 662.790985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.827477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.879154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 662.956943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.017511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.067260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 663.143635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.170595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.269861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 663.384814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.413546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.457112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 663.543358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.596574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.688080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 669.740794] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 22:05:04 (1788833104) [ 671.353578] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg647-server [ 672.921752] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 22:05:07 (1788833107) [ 674.434193] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg647-server [ 676.161147] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 22:05:10 (1788833110) [ 676.312433] LNet: 35835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.319473] LNet: 35835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 677.242513] Key type .llcrypt unregistered [ 677.245915] Key type ._llcrypt unregistered [ 683.412234] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 683.515322] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 683.519814] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 683.528510] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 686.158556] Key type ._llcrypt registered [ 686.160852] Key type .llcrypt registered [ 686.347522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 686.381510] alg: No test for adler32 (adler32-zlib) [ 687.305713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.363689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 687.397366] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 687.969264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 688.008616] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 689.650455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 689.680767] LNet: Removed LNI 10.1.2.3@tcp [ 689.787033] LNet: 36551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.798700] LNetError: 36551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 689.827310] LNet: Removed LNI 192.168.206.47@tcp [ 690.332890] Key type .llcrypt unregistered [ 690.337683] Key type ._llcrypt unregistered [ 701.293100] Key type ._llcrypt registered [ 701.294680] Key type .llcrypt registered [ 702.661585] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 22:05:37 (1788833137) [ 703.191384] Key type .llcrypt unregistered [ 703.195360] Key type ._llcrypt unregistered [ 709.042308] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 709.170090] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 709.174290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 709.179738] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 711.660485] Key type ._llcrypt registered [ 711.662677] Key type .llcrypt registered [ 711.832746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 711.855979] alg: No test for adler32 (adler32-zlib) [ 712.873821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.943567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 712.984619] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 713.688566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 713.710435] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 715.799293] LNet: 38044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.809560] LNetError: 38044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 715.817954] LNet: Removed LNI 10.1.2.3@tcp [ 715.832283] LNet: Removed LNI 192.168.206.47@tcp [ 716.301567] Key type .llcrypt unregistered [ 716.308802] Key type ._llcrypt unregistered [ 726.931894] Key type ._llcrypt registered [ 726.933953] Key type .llcrypt registered [ 728.154343] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 22:06:02 (1788833162) [ 728.547733] Key type .llcrypt unregistered [ 728.551959] Key type ._llcrypt unregistered [ 734.332668] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 734.541703] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 734.551937] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 734.565407] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 737.033271] Key type ._llcrypt registered [ 737.035432] Key type .llcrypt registered [ 737.211659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 737.234705] alg: No test for adler32 (adler32-zlib) [ 738.155635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.239828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 738.267867] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 740.373957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 740.410266] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 742.288797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 742.328713] LNet: Removed LNI 10.1.2.3@tcp [ 742.360290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 742.421231] LNet: Removed LNI 10.1.2.31@tcp [ 742.449203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 742.508202] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 743.252670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 743.270687] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 744.187818] LNet: 39957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.196980] LNetError: 39957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 744.207128] LNet: Removed LNI 10.1.2.31@tcp [ 744.213888] LNet: Removed LNI 10.1.2.3@tcp [ 744.866409] Key type .llcrypt unregistered [ 744.868035] Key type ._llcrypt unregistered [ 756.276712] Key type ._llcrypt registered [ 756.279841] Key type .llcrypt registered [ 757.754490] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 22:06:32 (1788833192) [ 758.116742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 758.142873] alg: No test for adler32 (adler32-zlib) [ 759.044835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.128374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 759.169382] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 760.094248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.115982] LNet: 40908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.122063] LNetError: 40908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 760.132575] LNet: Removed LNI 192.168.206.47@tcp [ 760.160577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.232435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.47@tcp [ 760.264552] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 760.980945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.47@tcp [ 761.021205] LNet: Removed LNI 192.168.206.47@tcp [ 761.123722] LNet: 41043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.885788] Key type .llcrypt unregistered [ 761.890840] Key type ._llcrypt unregistered [ 774.041283] Key type ._llcrypt registered [ 774.043964] Key type .llcrypt registered [ 775.457889] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 22:06:49 (1788833209) [ 777.186971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg647-server [ 778.940622] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 22:06:53 (1788833213) [ 779.550794] Key type .llcrypt unregistered [ 779.553914] Key type ._llcrypt unregistered [ 785.492336] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 785.627658] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 785.636800] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 785.647596] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 788.275301] Key type ._llcrypt registered [ 788.278718] Key type .llcrypt registered [ 788.460375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 788.485972] alg: No test for adler32 (adler32-zlib) [ 789.380388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.475953] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 790.167417] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 792.263562] LNet: 42882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.274516] LNetError: 42882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.285423] LNet: Removed LNI 192.168.206.47@tcp1 [ 792.293525] LNet: Removed LNI 10.1.2.3@tcp2 [ 792.859526] Key type .llcrypt unregistered [ 792.863569] Key type ._llcrypt unregistered [ 803.530953] Key type ._llcrypt registered [ 803.534305] Key type .llcrypt registered [ 804.920297] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 22:07:19 (1788833239) [ 805.337192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 805.355854] alg: No test for adler32 (adler32-zlib) [ 806.332764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.449886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 806.483224] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 807.302772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 807.316553] LNet: 43849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.324651] LNetError: 43849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.336302] LNet: Removed LNI 192.168.206.47@tcp [ 807.358376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.418051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 807.450105] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 813.785813] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 22:07:28 (1788833248) [ 813.952990] LNet: 44432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.965755] LNetError: 44432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.993208] LNet: Removed LNI 192.168.206.47@tcp [ 814.621348] Key type .llcrypt unregistered [ 814.625547] Key type ._llcrypt unregistered [ 820.862138] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 820.952846] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 820.961646] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 820.972380] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 823.550315] Key type ._llcrypt registered [ 823.552935] Key type .llcrypt registered [ 823.724254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 823.758704] alg: No test for adler32 (adler32-zlib) [ 824.697144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.782091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 824.820172] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 825.392497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 825.410684] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 827.448667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 827.480156] LNet: 45180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.487374] LNetError: 45180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 827.494610] LNet: Removed LNI 10.1.2.3@tcp [ 827.510095] LNet: Removed LNI 192.168.206.47@tcp [ 827.544384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.634849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 827.660103] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 827.665147] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 828.212757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 828.240918] LNet: 45207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.259320] LNetError: 45207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 828.268722] LNetError: 45207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 828.287134] LNet: Removed LNI 10.1.2.3@tcp [ 828.298403] LNet: Removed LNI 192.168.206.47@tcp [ 828.337171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.438532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 828.480452] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 828.490515] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 828.982186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 829.014550] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 829.910172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.927327] LNet: 45369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.932751] LNetError: 45369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 829.938676] LNetError: 45369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 829.942949] LNet: Removed LNI 10.1.2.3@tcp [ 829.944405] LNet: Removed LNI 192.168.206.47@tcp [ 829.952703] LNet: Removed LNI 192.168.206.47@tcp2 [ 829.988733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.073570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 830.099748] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 830.123583] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 830.133815] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 837.193858] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 22:07:52 (1788833272) [ 837.272871] LNet: 45957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.290415] LNetError: 45957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 837.298367] LNetError: 45957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 837.308846] LNet: Removed LNI 10.1.2.3@tcp [ 837.311646] LNet: Removed LNI 192.168.206.47@tcp [ 837.326431] LNet: Removed LNI 192.168.206.47@tcp2 [ 837.920926] Key type .llcrypt unregistered [ 837.927720] Key type ._llcrypt unregistered [ 844.554410] Key type ._llcrypt registered [ 844.557067] Key type .llcrypt registered [ 844.810357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 844.843619] alg: No test for adler32 (adler32-zlib) [ 845.801210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.897769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 845.918040] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 846.763698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.784605] LNet: 46358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.789836] LNetError: 46358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.803364] LNet: Removed LNI 192.168.206.47@tcp [ 846.827353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.890530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 846.913413] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 847.340040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.374180] LNet: 46388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.388726] LNetError: 46388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.414070] LNet: Removed LNI 192.168.206.47@tcp [ 847.436952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.561865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 847.600694] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 848.128898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 848.156494] LNet: Removed LNI 192.168.206.47@tcp [ 848.176321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.234986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 848.258128] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 848.791390] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 848.872471] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 848.876817] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 848.891485] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 851.283338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.315453] LNet: 46498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.320591] LNet: 46498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 851.333274] LNetError: 46498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.342544] LNetError: 46498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 851.360327] LNet: Removed LNI 192.168.206.47@tcp [ 851.393054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.469364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 851.494657] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 857.716322] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 22:08:12 (1788833292) [ 857.891536] LNet: 47092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.916031] LNetError: 47092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.940806] LNet: Removed LNI 192.168.206.47@tcp [ 858.570228] Key type .llcrypt unregistered [ 858.571983] Key type ._llcrypt unregistered [ 863.951543] Key type ._llcrypt registered [ 863.955922] Key type .llcrypt registered [ 864.128092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 864.140690] alg: No test for adler32 (adler32-zlib) [ 865.030116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.073249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 865.091923] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 865.780520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.812072] LNet: 47490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.821089] LNetError: 47490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.832057] LNet: Removed LNI 192.168.206.47@tcp [ 865.851299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.925624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 865.967337] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 873.233928] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 22:08:27 (1788833307) [ 873.312206] LNet: 48073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.322226] LNetError: 48073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.344726] LNet: Removed LNI 192.168.206.47@tcp [ 874.039484] Key type .llcrypt unregistered [ 874.041187] Key type ._llcrypt unregistered [ 880.529214] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 880.612849] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 880.617417] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 880.626580] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 883.329189] Key type ._llcrypt registered [ 883.331026] Key type .llcrypt registered [ 883.616413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 883.641625] alg: No test for adler32 (adler32-zlib) [ 884.541172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.603404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.644795] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 885.576793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.590827] LNet: 48526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.595689] LNetError: 48526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.607755] LNet: Removed LNI 192.168.206.47@tcp [ 885.631730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.683720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 885.713549] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 891.743215] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 22:08:46 (1788833326) [ 891.935978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.955992] LNet: 49124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.963009] LNetError: 49124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.971661] LNet: Removed LNI 192.168.206.47@tcp [ 892.016261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.124516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 897.705224] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 22:08:52 (1788833332) [ 897.823322] LNet: 49699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.421989] Key type .llcrypt unregistered [ 898.426521] Key type ._llcrypt unregistered [ 904.068553] Key type ._llcrypt registered [ 904.071339] Key type .llcrypt registered [ 904.319741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 904.341241] alg: No test for adler32 (adler32-zlib) [ 905.222140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.261321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 905.281957] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 906.073941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.114195] LNet: 50100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.126315] LNetError: 50100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.146795] LNet: Removed LNI 192.168.206.47@tcp [ 906.177669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.254629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 906.287550] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 912.304489] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 22:09:07 (1788833347) [ 912.397083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.424706] LNet: 50689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.431929] LNetError: 50689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.458114] LNet: Removed LNI 192.168.206.47@tcp [ 912.505765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.559649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 918.036279] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 22:09:12 (1788833352) [ 919.475965] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 921.037254] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 22:09:15 (1788833355) [ 922.757630] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 924.496916] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 22:09:19 (1788833359) [ 926.182672] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 927.935333] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 22:09:22 (1788833362) [ 930.096293] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 932.251183] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 22:09:26 (1788833366) [ 934.218402] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 936.434967] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 22:09:30 (1788833370) [ 936.590695] LNet: 52739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.326968] Key type .llcrypt unregistered [ 937.334657] Key type ._llcrypt unregistered [ 943.906243] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 944.041129] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 944.051469] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 944.054263] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 946.684404] Key type ._llcrypt registered [ 946.686396] Key type .llcrypt registered [ 946.931509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 946.948467] alg: No test for adler32 (adler32-zlib) [ 947.853620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.024727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 948.079927] LNet: Added LNI 192.168.206.47@tcp [8/512/0/180] [ 948.096639] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 948.098454] LNet: 53064:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 948.255607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.309520] LNet: 53083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.319498] LNetError: 53083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 948.332359] LNet: Removed LNI 10.1.2.3@tcp [ 948.339772] LNetError: 53083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.368340] LNet: Removed LNI 192.168.206.47@tcp [ 948.399921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.486923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 948.520633] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 948.594713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.609501] LNet: 53107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.612935] LNetError: 53107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.622962] LNet: Removed LNI 192.168.206.47@tcp [ 948.654851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.719928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 948.745904] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 948.751295] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 948.756359] LNet: 53115:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 948.838741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.859964] LNet: 53133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.866565] LNetError: 53133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 948.871422] LNet: Removed LNI 10.1.2.3@tcp [ 948.891518] LNetError: 53133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.905415] LNet: Removed LNI 192.168.206.47@tcp [ 948.920949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.964752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 948.979487] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 948.997753] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 949.000569] LNet: 53141:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 949.132598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.152559] LNet: 53167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.163198] LNetError: 53167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 949.167282] LNet: Removed LNI 10.1.2.3@tcp [ 949.170491] LNetError: 53167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.184738] LNet: Removed LNI 192.168.206.47@tcp [ 949.205548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.274877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 949.299188] LNet: Added LNI 192.168.206.47@tcp [8/256/0/182] [ 949.304096] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 949.399527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.424447] LNet: 53195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.430629] LNetError: 53195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 949.440704] LNet: Removed LNI 10.1.2.3@tcp [ 949.443459] LNetError: 53195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.466158] LNet: Removed LNI 192.168.206.47@tcp [ 949.485778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.588894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 949.621893] LNet: Added LNI 192.168.206.47@tcp [8/256/0/182] [ 949.626836] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 949.629837] LNet: 53203:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 949.757833] LNet: Removed LNI 10.1.2.3@tcp [ 949.773795] LNet: Removed LNI 192.168.206.47@tcp [ 950.427225] Key type .llcrypt unregistered [ 950.431039] Key type ._llcrypt unregistered [ 962.174702] Key type ._llcrypt registered [ 962.177971] Key type .llcrypt registered [ 963.324502] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 22:09:57 (1788833397) [ 964.726837] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 966.531677] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 22:10:01 (1788833401) [ 967.036537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 967.066788] alg: No test for adler32 (adler32-zlib) [ 968.006628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.067150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 968.093529] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 968.858632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.47 [ 968.918356] LNet: 54464:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 968.937668] LNet: Removed LNI 192.168.206.47@tcp [ 974.362633] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 22:10:09 (1788833409) [ 974.629524] LNet: 55034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.645622] LNetError: 55034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.663960] LNet: Removed LNI 192.168.206.47@tcp [ 975.247438] Key type .llcrypt unregistered [ 975.250855] Key type ._llcrypt unregistered [ 980.957638] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 981.250066] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 983.979418] Key type ._llcrypt registered [ 983.984810] Key type .llcrypt registered [ 984.263810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 984.306776] alg: No test for adler32 (adler32-zlib) [ 985.754096] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 992.666266] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 22:10:27 (1788833427) [ 992.783582] LNet: 55937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.794475] LNetError: 55937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 992.825862] LNet: Removed LNI 10.1.2.3@tcp [ 993.515592] Key type .llcrypt unregistered [ 993.522344] Key type ._llcrypt unregistered [ 999.297990] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 999.473088] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 999.478785] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 999.484988] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1001.049160] Key type ._llcrypt registered [ 1001.051272] Key type .llcrypt registered [ 1001.228840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.248585] alg: No test for adler32 (adler32-zlib) [ 1002.183711] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1008.610120] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 22:10:42 (1788833442) [ 1010.380172] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1012.136864] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 22:10:46 (1788833446) [ 1012.220035] LNet: 57109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.231604] LNetError: 57109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1012.270420] LNet: Removed LNI 10.1.2.3@tcp [ 1012.937749] Key type .llcrypt unregistered [ 1012.944820] Key type ._llcrypt unregistered [ 1018.760315] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1018.960396] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1018.962855] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1018.972798] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1021.646178] Key type ._llcrypt registered [ 1021.650254] Key type .llcrypt registered [ 1021.844643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1021.859433] alg: No test for adler32 (adler32-zlib) [ 1022.805793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1022.880460] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1031.310700] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 22:11:05 (1788833465) [ 1031.778383] LNet: 58344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.789530] LNetError: 58344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1031.815014] LNet: Removed LNI 10.1.2.3@tcp [ 1032.416680] Key type .llcrypt unregistered [ 1032.422885] Key type ._llcrypt unregistered [ 1038.548457] Key type ._llcrypt registered [ 1038.552812] Key type .llcrypt registered [ 1038.766625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1038.778588] alg: No test for adler32 (adler32-zlib) [ 1039.699091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1039.717784] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1046.490750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1047.079994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_interrupt -r 1 [ 1047.117184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_interrupt -r 1 [ 1047.154580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1047.182251] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1047.357619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1047.385613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1047.415900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1047.454471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1047.486686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1047.518560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1047.539604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1047.563514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1047.576652] LNet: 58779:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1047.607701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1047.641874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1047.673243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1047.715325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1047.753814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1047.801566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1047.851291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1048.340270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_dropped -r 1 [ 1048.388967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_dropped -r 1 [ 1048.427314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1048.445463] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1048.671498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1048.704266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1048.737220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1048.759960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1048.805880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1048.850504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1048.879260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1048.908843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1048.923232] LNet: 58888:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1048.933539] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1048.946638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1048.983427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1049.044961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1049.079738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1049.118718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1049.153607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1049.186694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1049.617625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_aborted -r 1 [ 1049.656887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_aborted -r 1 [ 1049.687144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1049.705707] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1049.907707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1049.935697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1049.962928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1049.995641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1050.040982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1050.091349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1050.112716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1050.141858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1050.167537] LNet: 58997:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1050.201120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1050.233191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1050.280697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1050.344254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1050.395464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1050.453334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1050.536878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1051.059811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_no_route -r 1 [ 1051.108814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_no_route -r 1 [ 1051.138909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1051.375943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1051.404401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1051.440262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1051.466410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1051.497321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1051.547711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1051.577352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1051.611565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1051.647964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1051.677497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1051.716234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1051.754426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1051.806969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1051.853700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1051.895367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1052.585191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_timeout -r 1 [ 1052.615629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_timeout -r 1 [ 1052.646810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1052.664463] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1052.669639] LNet: Skipped 2 previous similar messages [ 1052.902775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1052.941515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1052.982720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1053.020484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1053.074712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1053.118294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1053.150174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1053.203553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1053.232799] LNet: 59216:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1053.238434] LNet: 59216:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1053.255591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1053.302679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1053.354085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1053.401558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1053.457587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1053.506138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1053.551713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1054.068900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_error -r 1 [ 1054.131144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_error -r 1 [ 1054.179366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1054.455778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1054.499692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1054.542964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1054.590280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1054.617772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1054.646522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1054.699451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1054.740348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1054.782517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1054.837446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1054.887478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1054.918627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1054.955415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1054.990281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1055.017982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1055.266876] LNet: 59366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.271847] LNetError: 59366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1056.294331] LNet: Removed LNI 192.168.206.47@tcp [ 1056.812615] Key type .llcrypt unregistered [ 1056.815617] Key type ._llcrypt unregistered [ 1073.397846] Key type ._llcrypt registered [ 1073.399902] Key type .llcrypt registered [ 1074.551479] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 22:11:49 (1788833509) [ 1075.290078] Key type .llcrypt unregistered [ 1075.294377] Key type ._llcrypt unregistered [ 1081.483468] Key type ._llcrypt registered [ 1081.486047] Key type .llcrypt registered [ 1081.691338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1081.707401] alg: No test for adler32 (adler32-zlib) [ 1082.619900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1082.670691] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1090.875376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1099.315033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1099.358047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1099.382145] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1100.522196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_interrupt -r 1 [ 1100.570685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_interrupt -r 1 [ 1100.605456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_interrupt -r 1 [ 1100.639403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_interrupt -r 1 [ 1100.694619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_interrupt -r 1 [ 1100.731269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_interrupt -r 1 [ 1100.776680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_interrupt -r 1 [ 1100.831871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_interrupt -r 1 [ 1100.861839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1100.875646] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1100.881553] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1101.024293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1101.044803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1101.070112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.096644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1101.124735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1101.151221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1101.183719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1101.210872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1101.241661] LNet: 60828:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1101.262737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1101.289908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1101.321521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.349252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.384098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.429989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.476912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1102.105704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_dropped -r 1 [ 1102.153318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_dropped -r 1 [ 1102.198321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_dropped -r 1 [ 1102.234160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_dropped -r 1 [ 1102.279804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_dropped -r 1 [ 1102.327624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_dropped -r 1 [ 1102.365947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_dropped -r 1 [ 1102.403088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_dropped -r 1 [ 1102.442829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1102.459923] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1102.464909] LNet: Skipped 2 previous similar messages [ 1102.673754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1102.709857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1102.735957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1102.765955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.800581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.839354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1102.882215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1102.937680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1102.960982] LNet: 60949:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1102.986370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.012817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1103.043679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1103.075822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1103.106790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1103.134715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1103.165977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1103.741980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_aborted -r 1 [ 1103.779326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_aborted -r 1 [ 1103.837697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_aborted -r 1 [ 1103.906768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_aborted -r 1 [ 1103.955398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_aborted -r 1 [ 1104.013755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_aborted -r 1 [ 1104.099726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_aborted -r 1 [ 1104.129764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_aborted -r 1 [ 1104.163763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1104.203321] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1104.210955] LNet: Skipped 2 previous similar messages [ 1104.506116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.539499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.572222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.605633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.642789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.676249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.712235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.742957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.773227] LNet: 61070:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.797146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.835610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.863910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.897731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.934741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.961431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.994232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1105.551449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_no_route -r 1 [ 1105.624695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_no_route -r 1 [ 1105.669812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_no_route -r 1 [ 1105.743173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_no_route -r 1 [ 1105.794193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_no_route -r 1 [ 1105.835744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_no_route -r 1 [ 1105.879891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_no_route -r 1 [ 1105.911767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_no_route -r 1 [ 1105.953640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1106.180213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.214613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1106.261538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.288891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.353523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.388399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.435729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.461309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.511569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.558203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.609839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.649033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.704156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.763522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.797359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1107.333554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_timeout -r 1 [ 1107.392174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_timeout -r 1 [ 1107.470060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_timeout -r 1 [ 1107.511340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_timeout -r 1 [ 1107.547795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_timeout -r 1 [ 1107.576113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_timeout -r 1 [ 1107.635847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_timeout -r 1 [ 1107.671754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_timeout -r 1 [ 1107.704845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1107.735728] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1107.741757] LNet: Skipped 5 previous similar messages [ 1107.980721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.020374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1108.074430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.120806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.158688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.202546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.247876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.284026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.301666] LNet: 61313:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.310200] LNet: 61313:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1108.358658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.409342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.461801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.510598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.535544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.566584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.598888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1109.098469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_error -r 1 [ 1109.127588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_error -r 1 [ 1109.153893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_error -r 1 [ 1109.181417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_error -r 1 [ 1109.209995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_error -r 1 [ 1109.238922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_error -r 1 [ 1109.264592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_error -r 1 [ 1109.292822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_error -r 1 [ 1109.342124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1109.590905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.620250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.662849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.693394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.719784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.744337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.770413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.797324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.830864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.867572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.899808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.930530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.960253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.988174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.018562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1115.301522] LNet: 61510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.310329] LNetError: 61510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.318728] LNet: Removed LNI 192.168.206.47@tcp [ 1115.335230] LNet: Removed LNI 192.168.206.47@tcp1 [ 1115.736875] Key type .llcrypt unregistered [ 1115.740319] Key type ._llcrypt unregistered [ 1130.853263] Key type ._llcrypt registered [ 1130.855983] Key type .llcrypt registered [ 1132.128406] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 22:12:46 (1788833566) [ 1132.912446] Key type .llcrypt unregistered [ 1132.915205] Key type ._llcrypt unregistered [ 1138.335056] Key type ._llcrypt registered [ 1138.339714] Key type .llcrypt registered [ 1138.699353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1138.720827] alg: No test for adler32 (adler32-zlib) [ 1139.635146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1139.670367] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1146.337077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1146.955582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e remote_dropped -r 1 [ 1146.982798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e remote_dropped -r 1 [ 1147.012497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1147.028584] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1147.228256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1147.255418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1147.281603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1147.318966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1147.358586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1147.386812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1147.411245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1147.435154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1147.456303] LNet: 62752:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1147.474758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1147.505041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1147.543589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1147.571099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1147.601867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1147.644519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1147.686798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1148.182573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e remote_error -r 1 [ 1148.244953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e remote_error -r 1 [ 1148.289162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1148.327930] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1148.511855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1148.541640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1148.567750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1148.604043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1148.641973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1148.696962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1148.728785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1148.769606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1148.784269] LNet: 62861:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1148.811208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1148.852500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1148.890503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1148.912755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1148.952629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1148.983558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1149.015647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1149.511837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e remote_timeout -r 1 [ 1149.536779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e remote_timeout -r 1 [ 1149.569137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1149.582549] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1149.784706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1149.815056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1149.839217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1149.864682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1149.887384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1149.909728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1149.932692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1149.957788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1149.971119] LNet: 62971:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1149.986388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1150.017449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1150.039190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1150.061526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1150.093672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.134158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1150.171690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1150.402353] LNet: 63012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1150.409489] LNetError: 63012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1150.430730] LNet: Removed LNI 192.168.206.47@tcp [ 1150.764725] Key type .llcrypt unregistered [ 1150.767185] Key type ._llcrypt unregistered [ 1165.164900] Key type ._llcrypt registered [ 1165.167973] Key type .llcrypt registered [ 1166.206343] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 22:13:20 (1788833600) [ 1166.885681] Key type .llcrypt unregistered [ 1166.889932] Key type ._llcrypt unregistered [ 1172.608229] Key type ._llcrypt registered [ 1172.611677] Key type .llcrypt registered [ 1172.855825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1172.871463] alg: No test for adler32 (adler32-zlib) [ 1173.796497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1173.847888] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1181.061972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1189.082848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1189.114721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1189.128739] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1190.359391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e remote_dropped -r 1 [ 1190.393413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e remote_dropped -r 1 [ 1190.438089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e remote_dropped -r 1 [ 1190.460797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e remote_dropped -r 1 [ 1190.485916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e remote_dropped -r 1 [ 1190.513465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e remote_dropped -r 1 [ 1190.539416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e remote_dropped -r 1 [ 1190.563893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e remote_dropped -r 1 [ 1190.590513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1190.603799] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1190.608556] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.147@tcp [ 1190.784937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1190.821891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1190.861357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1190.889484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1190.915580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1190.940463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1190.965610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1190.996949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1191.019543] LNet: 64472:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1191.034457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1191.056362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1191.077690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1191.107634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1191.148728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1191.183226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1191.216181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1191.249714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1191.658912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e remote_error -r 1 [ 1191.686474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e remote_error -r 1 [ 1191.715888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e remote_error -r 1 [ 1191.757486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e remote_error -r 1 [ 1191.816622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e remote_error -r 1 [ 1191.850233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e remote_error -r 1 [ 1191.887936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e remote_error -r 1 [ 1191.958927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e remote_error -r 1 [ 1192.007475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1192.039915] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1192.050898] LNet: Skipped 2 previous similar messages [ 1192.257162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1192.282958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1192.319737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1192.345902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1192.378852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1192.407503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1192.459067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1192.492316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1192.511968] LNet: 64595:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1192.531806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1192.565434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1192.650978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1192.726989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.766369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.801045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1192.835959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1192.865827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.290613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e remote_timeout -r 1 [ 1193.334696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e remote_timeout -r 1 [ 1193.367682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e remote_timeout -r 1 [ 1193.402858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e remote_timeout -r 1 [ 1193.428968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e remote_timeout -r 1 [ 1193.456859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e remote_timeout -r 1 [ 1193.499098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e remote_timeout -r 1 [ 1193.533925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e remote_timeout -r 1 [ 1193.566421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1193.583870] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1193.773386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1193.807173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1193.856809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.892066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1193.918550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1193.940414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1193.961253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1193.985805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1193.999855] LNet: 64718:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1194.014902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1194.047955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1194.079972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1194.110739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1194.136775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1194.181719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1194.222775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1194.264694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.233403] LNet: 64796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.240594] LNetError: 64796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1198.246436] LNet: Removed LNI 192.168.206.47@tcp [ 1198.258611] LNet: Removed LNI 192.168.206.47@tcp1 [ 1198.659917] Key type .llcrypt unregistered [ 1198.661499] Key type ._llcrypt unregistered [ 1212.474427] Key type ._llcrypt registered [ 1212.477251] Key type .llcrypt registered [ 1213.518892] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 22:14:08 (1788833648) [ 1214.806981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1216.540968] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 22:14:11 (1788833651) [ 1217.338419] Key type .llcrypt unregistered [ 1217.344722] Key type ._llcrypt unregistered [ 1223.200233] Key type ._llcrypt registered [ 1223.201649] Key type .llcrypt registered [ 1223.480575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1223.505144] alg: No test for adler32 (adler32-zlib) [ 1224.514821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1224.559312] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1232.773986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1233.421461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e network_timeout -r 1 [ 1233.459174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e network_timeout -r 1 [ 1233.517727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1233.545904] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1233.782617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1233.815950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1233.825961] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1233.858374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1233.911342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1233.953586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1233.986647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1234.016804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1234.059486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1234.077906] LNet: 66334:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1234.096131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.131531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.161078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.194267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1234.230175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1234.273156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1234.602902] LNet: 66373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1234.612675] LNetError: 66373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1234.630431] LNet: Removed LNI 192.168.206.47@tcp [ 1234.948606] Key type .llcrypt unregistered [ 1234.954711] Key type ._llcrypt unregistered [ 1250.336203] Key type ._llcrypt registered [ 1250.338211] Key type .llcrypt registered [ 1250.500960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1250.510923] alg: No test for adler32 (adler32-zlib) [ 1251.411383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1251.443842] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1258.862241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1267.106060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1267.155453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1267.173636] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1268.661229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e network_timeout -r 1 [ 1268.694925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e network_timeout -r 1 [ 1268.725305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e network_timeout -r 1 [ 1268.753095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e network_timeout -r 1 [ 1268.778781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e network_timeout -r 1 [ 1268.804881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e network_timeout -r 1 [ 1268.848902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e network_timeout -r 1 [ 1268.880630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e network_timeout -r 1 [ 1268.917424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1268.974884] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1269.184032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1269.216279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1269.246874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1269.284506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1269.324229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1269.376821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1269.419602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1269.448061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1269.460496] LNet: 67246:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1269.477521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1269.532298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1269.558731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1269.604765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1269.645478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1269.692381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1273.935870] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1274.016307] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1274.265883] LNet: 67319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1274.280726] LNetError: 67319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.291637] LNet: Removed LNI 192.168.206.47@tcp [ 1274.302069] LNet: Removed LNI 192.168.206.47@tcp1 [ 1274.682761] Key type .llcrypt unregistered [ 1274.686397] Key type ._llcrypt unregistered [ 1289.779359] Key type ._llcrypt registered [ 1289.782135] Key type .llcrypt registered [ 1291.064984] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 22:15:25 (1788833725) [ 1291.474660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1291.488905] alg: No test for adler32 (adler32-zlib) [ 1292.395819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1292.441994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1292.475561] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1293.033322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1293.063194] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1293.854823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1293.917710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1293.962432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1294.006458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1294.042374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1294.112978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1294.165932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.47@tcp [ 1294.200375] LNetError: 68415:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.47@tcp to recovery [ 1294.495313] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1294.509242] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1296.546349] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1297.715162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1300.640717] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1301.317651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1301.360328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1301.428961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1301.448350] LNet: 68522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.454184] LNetError: 68522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1301.460300] LNet: Removed LNI 192.168.206.47@tcp [ 1301.477798] LNet: Removed LNI 192.168.206.47@tcp1 [ 1301.503992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1301.577874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1301.618152] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1302.456667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1302.492319] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1303.441487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1303.470604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1303.525695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1303.571657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1303.641845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1303.704629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1303.749865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1303.798733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.47@tcp [ 1303.837706] LNetError: 68795:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.47@tcp to recovery [ 1304.607538] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1307.481855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1310.751455] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1310.762584] LNet: Skipped 1 previous similar message [ 1315.661119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1315.709890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1315.739299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1322.686060] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 22:15:57 (1788833757) [ 1322.839583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1322.865228] LNet: 69513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1322.879948] LNetError: 69513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1322.888288] LNetError: 69513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1322.896079] LNet: Removed LNI 192.168.206.47@tcp [ 1322.909107] LNet: Removed LNI 192.168.206.47@tcp1 [ 1322.942712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1323.023299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1323.061239] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1323.785817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1323.810083] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1324.487420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1324.557766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1324.595230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1324.636480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1324.674777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1324.714882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1324.755733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.47@tcp --health 0 [ 1324.862863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1325.023410] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.47@tcp [ 1325.024573] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1325.031830] LNet: Skipped 1 previous similar message [ 1339.488130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1339.691258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1339.717526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1339.747853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.47@tcp --health 500 [ 1339.872590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1346.490373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1346.581137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1346.609880] LNet: 69997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1346.617840] LNetError: 69997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1346.623604] LNetError: 69997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1346.638128] LNet: Removed LNI 192.168.206.47@tcp [ 1346.658236] LNet: Removed LNI 192.168.206.47@tcp1 [ 1346.677129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1346.739879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1346.760614] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1347.374466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1347.395897] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1348.047639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1348.083174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1348.135681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1348.165184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1348.195847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1348.225819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1348.257484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1348.283126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.47@tcp --health 0 [ 1349.791446] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1349.802631] LNet: Skipped 3 previous similar messages [ 1350.735595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1359.931564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1359.984426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1360.033413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1365.304580] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 22:16:40 (1788833800) [ 1365.628857] LNet: 71045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.633689] LNetError: 71045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1365.639635] LNetError: 71045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1365.649666] LNet: Removed LNI 192.168.206.47@tcp [ 1365.660199] LNet: Removed LNI 192.168.206.47@tcp1 [ 1366.151388] Key type .llcrypt unregistered [ 1366.154920] Key type ._llcrypt unregistered [ 1370.715645] Key type ._llcrypt registered [ 1370.720766] Key type .llcrypt registered [ 1370.914135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1370.930396] alg: No test for adler32 (adler32-zlib) [ 1371.840748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1371.870053] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1379.245926] Lustre: DEBUG MARKER: Initial discovery [ 1379.393874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.147@tcp [ 1381.633555] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1381.764953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1381.835646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.147@tcp [ 1381.856087] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1381.892547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1381.926402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.147@tcp -r 1 -m GET -l 3 [ 1381.960211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.147@tcp -r 1 -m GET -e local_error [ 1381.989770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.147@tcp -r 1 -m PUT -l 6 [ 1382.112297] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1383.312539] Lustre: DEBUG MARKER: Force oleg647-client.virtnet to discover 192.168.206.147@tcp (in background) [ 1383.469154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.147@tcp [ 1384.872504] Lustre: DEBUG MARKER: Force oleg647-server to discover 192.168.206.47@tcp [ 1386.527630] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1387.376850] Lustre: DEBUG MARKER: Wait for 71801 [ 1388.983271] Lustre: DEBUG MARKER: Finished wait on 71801 [ 1389.035619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1389.072924] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1389.084083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1389.364414] LNet: 72252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.370108] LNetError: 72252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.384648] LNet: Removed LNI 192.168.206.47@tcp [ 1389.740649] Key type .llcrypt unregistered [ 1389.743484] Key type ._llcrypt unregistered [ 1402.872344] Key type ._llcrypt registered [ 1402.875149] Key type .llcrypt registered [ 1404.254893] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 22:17:18 (1788833838) [ 1404.718523] Key type .llcrypt unregistered [ 1404.721447] Key type ._llcrypt unregistered [ 1410.536946] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1410.712682] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1410.717174] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1410.733322] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1413.278611] Key type ._llcrypt registered [ 1413.284341] Key type .llcrypt registered [ 1413.456793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1413.467224] alg: No test for adler32 (adler32-zlib) [ 1414.417418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1414.486307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1414.531607] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1415.240965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1415.254282] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1422.533588] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 22:17:37 (1788833857) [ 1422.612443] LNet: 74312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.617635] LNetError: 74312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1422.637600] LNet: Removed LNI 10.1.2.3@tcp [ 1422.651880] LNet: Removed LNI 192.168.206.47@tcp [ 1423.389530] Key type .llcrypt unregistered [ 1423.393045] Key type ._llcrypt unregistered [ 1429.894791] Key type ._llcrypt registered [ 1429.897795] Key type .llcrypt registered [ 1430.092180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1430.128164] alg: No test for adler32 (adler32-zlib) [ 1431.134510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.206672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1431.238163] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1432.064164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1432.078623] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1433.163407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.47@tcp [ 1434.271279] LNetError: Unexpected error -22 connecting to 192.168.206.47@tcp at host 192.168.206.47:988 [ 1438.813183] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 22:17:53 (1788833873) [ 1438.891883] LNet: 75443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.898468] LNetError: 75443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1438.903538] LNet: Removed LNI 10.1.2.3@tcp [ 1438.917437] LNet: Removed LNI 192.168.206.47@tcp [ 1439.492716] Key type .llcrypt unregistered [ 1439.496703] Key type ._llcrypt unregistered [ 1444.480680] Key type ._llcrypt registered [ 1444.482213] Key type .llcrypt registered [ 1444.694900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1444.715624] alg: No test for adler32 (adler32-zlib) [ 1445.606536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1445.672177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1445.704115] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1446.287873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1446.306142] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 1447.059211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.47@tcp1 --nid 192.168.206.47@tcp2 [ 1450.781582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp1 192.168.206.47@tcp2 [ 1450.815307] LNetError: 76283:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.47@tcp1 to 192.168.206.47@tcp2 - Route table may be misconfigured [ 1450.850903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp2 192.168.206.47@tcp1 [ 1456.302335] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 22:18:11 (1788833891) [ 1456.562069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1456.599616] LNet: 76855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1456.603966] LNetError: 76855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1456.613460] LNet: Removed LNI 192.168.206.47@tcp1 [ 1457.638830] LNet: Removed LNI 192.168.206.47@tcp2 [ 1457.674371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.742332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1457.768551] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1458.457436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1458.478905] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1459.379324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1459.418305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp 192.168.206.47@tcp [ 1459.460946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp1 192.168.206.47@tcp1 [ 1459.497533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.47@tcp -d 192.168.206.47@tcp -e network_timeout [ 1459.548635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -e network_timeout [ 1459.588902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -e network_timeout [ 1459.636386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -e network_timeout [ 1459.673350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1459.692501] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1459.709822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1459.744748] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1459.812902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1459.913105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1465.949627] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 22:18:20 (1788833900) [ 1466.046717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1466.080185] LNet: 77712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.099830] LNetError: 77712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.107447] LNetError: 77712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1466.124353] LNet: Removed LNI 192.168.206.47@tcp [ 1467.174629] LNet: Removed LNI 192.168.206.47@tcp1 [ 1467.217410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1467.306858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1467.557106] LNet: 77754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1468.023076] Key type .llcrypt unregistered [ 1468.029416] Key type ._llcrypt unregistered [ 1477.744393] Key type ._llcrypt registered [ 1477.746221] Key type .llcrypt registered [ 1478.888968] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 22:18:33 (1788833913) [ 1479.163517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1479.179770] alg: No test for adler32 (adler32-zlib) [ 1480.067411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.337705] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1482.805980] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 22:18:37 (1788833917) [ 1482.878168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1482.894453] LNet: 78845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.917873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1482.971242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1482.994381] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1483.456452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1483.473505] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1484.093609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1484.132254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp1 [ 1484.165857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp1 [ 1488.995340] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 22:18:43 (1788833923) [ 1490.433881] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg647-server [ 1491.938213] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 22:18:46 (1788833926) [ 1493.461117] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg647-server [ 1494.988759] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 22:18:49 (1788833929) [ 1496.577382] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg647-server [ 1497.940918] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 22:18:52 (1788833932) [ 1499.377606] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg647-server [ 1500.648549] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 22:18:55 (1788833935) [ 1502.123418] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg647-server [ 1503.334792] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 22:18:58 (1788833938) [ 1504.529911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg647-server [ 1505.909814] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 22:19:00 (1788833940) [ 1507.524507] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg647-server [ 1508.706484] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 22:19:03 (1788833943) [ 1510.096995] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg647-server [ 1511.409367] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 22:19:06 (1788833946) [ 1513.189598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg647-server [ 1514.899934] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 22:19:09 (1788833949) [ 1514.971156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1514.999491] LNet: 82720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1515.005148] LNetError: 82720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1515.013329] LNet: Removed LNI 192.168.206.47@tcp [ 1516.072270] LNet: Removed LNI 192.168.206.47@tcp1 [ 1516.098506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.173783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1516.207837] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1516.863142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1516.931718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1517.084398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.114639] LNet: 82873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.125286] LNetError: 82873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.134314] LNetError: 82873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1517.148765] LNet: Removed LNI 192.168.206.47@tcp [ 1517.177831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.255842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1517.279750] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1517.996811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1518.197612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1518.223817] LNetError: 83018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1519.274735] LNet: Removed LNI 192.168.206.47@tcp [ 1519.302713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.343305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1519.372385] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1520.166724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1520.267513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1520.486799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1520.525452] LNet: 83165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1520.534909] LNet: 83165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1520.544488] LNetError: 83165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1520.561157] LNet: Removed LNI 192.168.206.47@tcp [ 1520.587395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1520.693140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1520.716401] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1521.371663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1521.607756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1522.661153] LNet: Removed LNI 192.168.206.47@tcp [ 1522.698711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.770693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1522.797053] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1523.488793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1523.563756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1523.805957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1524.906194] LNet: Removed LNI 192.168.206.47@tcp [ 1524.956761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.036444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1525.078320] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1525.763648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1525.966809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1525.985879] LNet: 83603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.990884] LNet: 83603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1525.997418] LNetError: 83603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.009455] LNetError: 83603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1526.029450] LNet: Removed LNI 192.168.206.47@tcp [ 1526.062622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1526.103542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1526.120926] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1526.643833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1526.708278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1526.886601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1527.978539] LNet: Removed LNI 192.168.206.47@tcp [ 1528.001675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.050848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1528.084604] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1528.787458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1529.061262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1530.153977] LNet: Removed LNI 192.168.206.47@tcp [ 1530.183326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.248933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1530.282035] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1530.828710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1530.914933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1531.174109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1532.261445] LNet: Removed LNI 192.168.206.47@tcp [ 1532.292216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1532.351833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1532.370329] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1533.052603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1533.299414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.375655] LNet: Removed LNI 192.168.206.47@tcp [ 1534.424632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.504185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1534.538631] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1535.286395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1535.410975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1535.723252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.748501] LNet: 84335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.763418] LNet: 84335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1535.773149] LNetError: 84335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.782320] LNetError: 84335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1535.811428] LNet: Removed LNI 192.168.206.47@tcp [ 1535.847221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.914408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1535.934112] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1536.765800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1537.143742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.215789] LNet: Removed LNI 192.168.206.47@tcp [ 1538.270615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.344969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1538.378657] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1539.183144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1539.234634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.288969] LNet: Removed LNI 192.168.206.47@tcp [ 1539.313773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.360740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1539.382291] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1540.202436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1540.326146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.380105] LNet: Removed LNI 192.168.206.47@tcp [ 1540.399843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.468879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1540.542890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.576560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.626557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1540.721716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.770138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.842659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1540.941781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.973791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.013588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1541.033754] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1541.607208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1541.636688] LNet: Removed LNI 192.168.206.47@tcp [ 1541.651785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.718277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1547.700833] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 22:19:42 (1788833982) [ 1547.816712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1547.860399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.910981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1547.928398] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/180] [ 1548.961795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.996615] LNet: Removed LNI 192.168.206.47@tcp231 [ 1549.021308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.112619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1549.156908] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/180] [ 1549.167915] LNet: 85648:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1549.418726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1549.452910] LNet: Removed LNI 192.168.206.47@tcp231 [ 1549.468757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1549.492325] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/180] [ 1550.530253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.564094] LNet: Removed LNI 192.168.206.47@tcp231 [ 1550.582710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.637288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1550.663884] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/0] [ 1550.669713] LNet: 85800:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1551.053146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1551.087672] LNet: Removed LNI 192.168.206.47@tcp231 [ 1551.106712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1551.132515] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/0] [ 1552.045938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.068281] LNet: 85944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.078511] LNet: 85944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1552.084281] LNetError: 85944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.090468] LNetError: 85944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1552.106666] LNet: Removed LNI 192.168.206.47@tcp231 [ 1552.137401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.199815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1552.232114] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/60] [ 1552.240764] LNet: 85952:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1552.544779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1552.581325] LNet: Removed LNI 192.168.206.47@tcp231 [ 1552.607455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1552.640794] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/60] [ 1553.534367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1553.565148] LNet: Removed LNI 192.168.206.47@tcp231 [ 1553.601374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1553.697509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1553.738207] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/180] [ 1554.079857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1554.108986] LNet: Removed LNI 192.168.206.47@tcp231 [ 1554.123521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1554.147845] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/180] [ 1560.150491] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 22:19:54 (1788833994) [ 1560.294128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.329506] LNet: Removed LNI 192.168.206.47@tcp231 [ 1560.350685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.409382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1560.430486] LNet: Added LNI 192.168.206.47@tcp232 [8/256/0/180] [ 1560.968457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1561.001282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1561.027600] LNet: Removed LNI 192.168.206.47@tcp232 [ 1561.038557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1561.061621] LNet: Added LNI 192.168.206.47@tcp232 [8/256/0/180] [ 1561.659294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1561.690756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1561.721154] LNet: Removed LNI 192.168.206.47@tcp232 [ 1561.739981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1561.757450] LNet: Added LNI 192.168.206.47@tcp232 [8/256/0/180] [ 1562.319630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1562.352679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1562.397495] LNet: Removed LNI 192.168.206.47@tcp232 [ 1562.417346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1562.438603] LNet: Added LNI 192.168.206.47@tcp232 [8/256/0/180] [ 1563.089400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1563.127450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1563.156322] LNet: Removed LNI 192.168.206.47@tcp232 [ 1563.168317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1568.076891] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 22:20:02 (1788834002) [ 1568.911125] Key type .llcrypt unregistered [ 1568.915712] Key type ._llcrypt unregistered [ 1573.678563] Key type ._llcrypt registered [ 1573.681451] Key type .llcrypt registered [ 1573.836111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1573.858496] alg: No test for adler32 (adler32-zlib) [ 1574.779060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1574.813843] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1581.091755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1588.590343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.625444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1588.639856] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1589.878458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_interrupt -r 2 [ 1589.919563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_interrupt -r 2 [ 1589.943407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_interrupt -r 2 [ 1589.974628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_interrupt -r 2 [ 1590.017440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_interrupt -r 2 [ 1590.047321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_interrupt -r 2 [ 1590.079334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_interrupt -r 2 [ 1590.110269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_interrupt -r 2 [ 1590.152567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp 192.168.206.147@tcp [ 1590.176648] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1590.186700] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1590.382282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1590.416378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1590.452599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1590.479932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1590.507404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1590.533656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1590.572734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1590.594717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1590.619525] LNet: 88597:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1590.641210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1590.665908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1590.690903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1590.732840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1590.763903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1590.797854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1591.200328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_dropped -r 2 [ 1591.231782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_dropped -r 2 [ 1591.262070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_dropped -r 2 [ 1591.301519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_dropped -r 2 [ 1591.331417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_dropped -r 2 [ 1591.360851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_dropped -r 2 [ 1591.383857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_dropped -r 2 [ 1591.408501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_dropped -r 2 [ 1591.434989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp 192.168.206.147@tcp [ 1591.459654] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1591.702646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1591.743754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1591.778863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1591.808249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1591.843421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1591.876189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1591.905778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1591.927893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1591.943748] LNet: 88717:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1591.957354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1591.983795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1592.010803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1592.033921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1592.056785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1592.079621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1592.495647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_aborted -r 2 [ 1592.528418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_aborted -r 2 [ 1592.560863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_aborted -r 2 [ 1592.591413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_aborted -r 2 [ 1592.626617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_aborted -r 2 [ 1592.666684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_aborted -r 2 [ 1592.704821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_aborted -r 2 [ 1592.735269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_aborted -r 2 [ 1592.772645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp 192.168.206.147@tcp [ 1592.791652] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1592.995169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1593.028600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1593.060168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1593.106927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1593.145851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1593.177365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1593.202853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1593.228475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1593.247574] LNet: 88838:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1593.269640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1593.297278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1593.320835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1593.343118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1593.369568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1593.407960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1593.755758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_no_route -r 2 [ 1593.796835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_no_route -r 2 [ 1593.844871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_no_route -r 2 [ 1593.889847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_no_route -r 2 [ 1593.921943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_no_route -r 2 [ 1593.961871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_no_route -r 2 [ 1594.001556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_no_route -r 2 [ 1594.035780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_no_route -r 2 [ 1594.083426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp 192.168.206.147@tcp [ 1594.292621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1594.313846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1594.331741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1594.353213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1594.381958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1594.408034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1594.433305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1594.466869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1594.499614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1594.538344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1594.575923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1594.627947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1594.659602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1594.692663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1595.099436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_timeout -r 2 [ 1595.130469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_timeout -r 2 [ 1595.162819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_timeout -r 2 [ 1595.192940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_timeout -r 2 [ 1595.219384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_timeout -r 2 [ 1595.253579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_timeout -r 2 [ 1595.286924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_timeout -r 2 [ 1595.317829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_timeout -r 2 [ 1595.353756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp 192.168.206.147@tcp [ 1595.370897] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1595.378634] LNet: Skipped 1 previous similar message [ 1595.594377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1595.640544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1595.674281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1595.710637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1595.746915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1595.788894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1595.813652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1595.853944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1595.877441] LNet: 89078:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1595.886571] LNet: 89078:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1595.903774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1595.921319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1595.945744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1595.973270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1596.006929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1596.032902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1596.468734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e remote_dropped -r 2 [ 1596.503462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e remote_dropped -r 2 [ 1596.544643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e remote_dropped -r 2 [ 1596.590811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e remote_dropped -r 2 [ 1596.634104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e remote_dropped -r 2 [ 1596.664917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e remote_dropped -r 2 [ 1596.695590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e remote_dropped -r 2 [ 1596.717981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e remote_dropped -r 2 [ 1596.750419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp 192.168.206.147@tcp [ 1596.772930] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.147@tcp [ 1596.983249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1597.009838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1597.036254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1597.062941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1597.098247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1597.130770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1597.162148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1597.197644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1597.228309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1597.260492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1597.283695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1597.314213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1597.342852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1597.379699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1601.475338] LNet: 89272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.481504] LNetError: 89272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.486657] LNet: Removed LNI 192.168.206.47@tcp [ 1601.498166] LNet: Removed LNI 192.168.206.47@tcp1 [ 1601.817807] Key type .llcrypt unregistered [ 1601.821274] Key type ._llcrypt unregistered [ 1613.498793] Key type ._llcrypt registered [ 1613.502789] Key type .llcrypt registered [ 1614.479730] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 22:20:49 (1788834049) [ 1614.795977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1614.817261] alg: No test for adler32 (adler32-zlib) [ 1615.719622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.783294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1615.804960] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1616.497959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1616.525510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.47@tcp [ 1616.543368] LNetError: 90247:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.47@tcp to recovery [ 1616.619825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.47@tcp [ 1616.696431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1621.541721] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 22:20:56 (1788834056) [ 1621.634884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.658670] LNet: 90833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.665833] LNetError: 90833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1621.680705] LNet: Removed LNI 192.168.206.47@tcp [ 1621.719445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.767818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1621.792778] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1622.356296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1622.368470] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1623.082257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.47@tcp [ 1623.096542] LNetError: 91102:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.47@tcp to recovery [ 1623.775469] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1623.776754] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1624.309608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.47@tcp [ 1631.876408] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 22:21:06 (1788834066) [ 1633.295747] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg647-server [ 1634.955136] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 22:21:09 (1788834069) [ 1636.520445] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg647-server [ 1638.216889] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 22:21:12 (1788834072) [ 1639.819841] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg647-server [ 1641.735242] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 22:21:16 (1788834076) [ 1641.869400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.917180] LNet: 92728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.924986] LNetError: 92728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1641.931252] LNet: Removed LNI 192.168.206.47@tcp [ 1641.955848] LNet: Removed LNI 192.168.206.47@tcp1 [ 1642.011689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.064107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1642.083772] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1642.805592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1642.888140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1642.958641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1643.068624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1643.103956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1643.191192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1648.774533] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 22:21:23 (1788834083) [ 1648.844416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.863674] LNet: 93458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.870487] LNetError: 93458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1648.875729] LNetError: 93458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1648.891908] LNet: Removed LNI 192.168.206.47@tcp [ 1648.917116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.971888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1648.996293] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1655.498718] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 22:21:30 (1788834090) [ 1655.800271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.854270] LNet: 94167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.868690] LNetError: 94167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.887556] LNet: Removed LNI 192.168.206.47@tcp [ 1655.904887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.959727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1655.987932] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1661.636950] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 22:21:36 (1788834096) [ 1662.985222] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1664.399917] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 22:21:39 (1788834099) [ 1664.841310] LNet: 95201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1664.851423] LNetError: 95201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.864957] LNet: Removed LNI 192.168.206.47@tcp [ 1665.574304] Key type .llcrypt unregistered [ 1665.578574] Key type ._llcrypt unregistered [ 1670.715545] Key type ._llcrypt registered [ 1670.718926] Key type .llcrypt registered [ 1670.933205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1670.956764] alg: No test for adler32 (adler32-zlib) [ 1671.879378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1671.901608] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1677.463630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1682.345705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.147@tcp [ 1682.672219] LNet: 95615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.679929] LNetError: 95615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1682.693011] LNet: Removed LNI 192.168.206.47@tcp [ 1683.178651] Key type .llcrypt unregistered [ 1683.182447] Key type ._llcrypt unregistered [ 1692.562279] Key type ._llcrypt registered [ 1692.564158] Key type .llcrypt registered [ 1693.626215] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1694.880726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1696.283330] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 22:22:11 (1788834131) [ 1697.603717] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1699.005952] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 22:22:13 (1788834133) [ 1700.437197] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg647-server [ 1701.573260] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 22:22:16 (1788834136) [ 1702.948272] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg647-server [ 1704.546295] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 22:22:19 (1788834139) [ 1705.629160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1705.642309] alg: No test for adler32 (adler32-zlib) [ 1706.679903] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 1706.888740] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1708.535254] Key type lgssc registered [ 1709.583685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1726.859206] Key type lgssc unregistered [ 1727.116620] LNet: 98783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.129980] LNetError: 98783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.148628] LNet: Removed LNI 192.168.206.47@tcp [ 1727.928808] Key type .llcrypt unregistered [ 1727.930588] Key type ._llcrypt unregistered [ 1741.857732] Key type ._llcrypt registered [ 1741.860532] Key type .llcrypt registered [ 1743.171982] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 22:22:57 (1788834177) [ 1743.592864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1743.608778] alg: No test for adler32 (adler32-zlib) [ 1744.516098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.582828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1744.627928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1744.675912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1744.730454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1751.487383] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 22:23:06 (1788834186) [ 1751.597495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.622691] LNet: 100192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1751.656480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.694836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1751.711049] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1752.423832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.447865] LNet: 100339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.456802] LNetError: 100339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1752.482772] LNet: Removed LNI 192.168.206.47@tcp [ 1752.513334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.574348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1752.603669] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1753.175252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.208286] LNetError: 100367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.224550] LNet: Removed LNI 192.168.206.47@tcp [ 1753.257413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.316060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1753.350604] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1759.253412] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 22:23:14 (1788834194) [ 1759.390442] LNet: 100953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.399805] LNet: 100953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1759.408855] LNetError: 100953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.426840] LNet: Removed LNI 192.168.206.47@tcp [ 1760.005320] Key type .llcrypt unregistered [ 1760.010871] Key type ._llcrypt unregistered [ 1765.403523] Key type ._llcrypt registered [ 1765.411488] Key type .llcrypt registered [ 1765.617631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1765.634748] alg: No test for adler32 (adler32-zlib) [ 1766.537477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.594203] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1766.596186] LNet: *** cfs_fail_loc=e003, val=100*** [ 1767.245780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1767.258628] LNet: *** cfs_fail_loc=e003, val=100*** [ 1767.260590] LNet: Skipped 2 previous similar messages [ 1767.265779] Ping reply received with mapping: [ 1767.267605] found metadata reply NI[192.168.206.47@tcp] [ 1767.269912] 2f ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1767.272712] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1767.275308] 5a 5a 5a 5a 5a 5a 5a 5a [ 1767.369766] LNet: 101351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.379290] LNetError: 101351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.389286] LNet: Removed LNI 192.168.206.47@tcp [ 1767.972109] Key type .llcrypt unregistered [ 1767.981197] Key type ._llcrypt unregistered [ 1777.921789] Key type ._llcrypt registered [ 1777.923613] Key type .llcrypt registered [ 1778.956433] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 22:23:33 (1788834213) [ 1779.354506] Key type .llcrypt unregistered [ 1779.358534] Key type ._llcrypt unregistered [ 1784.966673] Key type ._llcrypt registered [ 1784.968852] Key type .llcrypt registered [ 1785.202087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1785.235125] alg: No test for adler32 (adler32-zlib) [ 1786.498332] LNetError: 102387:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1786.992580] Key type .llcrypt unregistered [ 1786.994064] Key type ._llcrypt unregistered [ 1791.534272] Key type ._llcrypt registered [ 1791.536540] Key type .llcrypt registered [ 1792.573327] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 22:23:47 (1788834227) [ 1793.108740] Key type .llcrypt unregistered [ 1793.110498] Key type ._llcrypt unregistered [ 1798.891165] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1798.894614] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1801.422249] Key type ._llcrypt registered [ 1801.426621] Key type .llcrypt registered [ 1801.599761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1801.618737] alg: No test for adler32 (adler32-zlib) [ 1802.564189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.636565] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1811.112509] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 22:24:05 (1788834245) [ 1811.217635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.267274] LNet: 104152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1811.281664] LNetError: 104152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1811.307735] LNet: Removed LNI 10.1.2.3@tcp [ 1811.349313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.090561] Key type .llcrypt unregistered [ 1812.092714] Key type ._llcrypt unregistered [ 1818.295478] Key type ._llcrypt registered [ 1818.299313] Key type .llcrypt registered [ 1818.463693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1818.482377] alg: No test for adler32 (adler32-zlib) [ 1819.408333] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1819.471363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1819.628186] LNet: 104445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.641341] LNetError: 104445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1819.666603] LNet: Removed LNI 192.168.206.47@tcp [ 1820.261493] Key type .llcrypt unregistered [ 1820.266644] Key type ._llcrypt unregistered [ 1830.858979] Key type ._llcrypt registered [ 1830.860353] Key type .llcrypt registered [ 1832.076035] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 22:24:26 (1788834266) [ 1832.683382] Key type .llcrypt unregistered [ 1832.687393] Key type ._llcrypt unregistered [ 1838.444640] Key type ._llcrypt registered [ 1838.447765] Key type .llcrypt registered [ 1838.622438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1838.634299] alg: No test for adler32 (adler32-zlib) [ 1845.261538] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 22:24:39 (1788834279) [ 1845.333293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1845.367325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1845.428618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1845.462456] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1846.301224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1846.335255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1846.363946] LNet: Removed LNI 192.168.206.47@tcp [ 1846.643266] LNet: 106183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1847.097461] Key type .llcrypt unregistered [ 1847.106928] Key type ._llcrypt unregistered [ 1857.521592] Key type ._llcrypt registered [ 1857.525536] Key type .llcrypt registered [ 1858.970188] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 22:24:53 (1788834293) [ 1859.393554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1859.412233] alg: No test for adler32 (adler32-zlib) [ 1860.331910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1860.390706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1860.420042] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1861.000591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1861.054480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.47@tcp [ 1861.087371] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1861.090912] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1861.093964] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1861, 0, 0 [ 1861.098425] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1861.100675] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1861.111090] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1861.114394] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1861, 0, 0 [ 1861.118124] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1861.119379] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1861.123813] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1861.130351] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1861, 0, 1911 [ 1861.135182] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1861.138869] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1861.142280] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1861.145708] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1861, 0, 1911 [ 1861.152516] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1866.435864] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 22:25:01 (1788834301) [ 1866.812869] LNet: 107720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.819599] LNetError: 107720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1867.884862] LNet: Removed LNI 192.168.206.47@tcp [ 1868.552492] Key type .llcrypt unregistered [ 1868.557883] Key type ._llcrypt unregistered [ 1874.458574] Key type ._llcrypt registered [ 1874.461870] Key type .llcrypt registered [ 1874.645150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1874.655471] alg: No test for adler32 (adler32-zlib) [ 1875.592086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1875.659928] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1882.781978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1890.937446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1890.979334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1890.998838] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1891.978547] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1892.089919] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1892.093837] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1892.098757] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1894.469540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1894.498246] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1904.696170] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 22:25:39 (1788834339) [ 1904.777525] LNet: 109217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.789747] LNetError: 109217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1904.796891] LNet: Removed LNI 192.168.206.47@tcp [ 1905.823159] LNet: Removed LNI 192.168.206.47@tcp1 [ 1905.825137] LNetError: 109217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1905.837035] LNetError: 109217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1905.867223] LNet: Removed LNI 10.1.2.3@tcp99 [ 1906.492091] Key type .llcrypt unregistered [ 1906.494811] Key type ._llcrypt unregistered [ 1912.931764] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1912.937223] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1915.453863] Key type ._llcrypt registered [ 1915.457154] Key type .llcrypt registered [ 1915.605204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1915.626136] alg: No test for adler32 (adler32-zlib) [ 1916.510296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.596988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1916.645755] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1917.276418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1917.298564] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1919.269267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1919.301476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1919.335817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1919.357494] LNetError: 109967:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1919.405353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1919.461860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1919.895229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1919.970217] LNet: 109987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.976837] LNetError: 109987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1919.981968] LNet: Removed LNI 10.1.2.3@tcp [ 1920.998873] LNet: Removed LNI 192.168.206.47@tcp [ 1921.646442] Key type .llcrypt unregistered [ 1921.652265] Key type ._llcrypt unregistered [ 1932.596769] Key type ._llcrypt registered [ 1932.600690] Key type .llcrypt registered [ 1933.796199] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 22:26:08 (1788834368) [ 1934.222685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1934.236852] alg: No test for adler32 (adler32-zlib) [ 1935.147772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1935.230294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1935.253180] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1935.922326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1936.041711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg647-client@tcp [ 1942.735340] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 22:26:17 (1788834377) [ 1942.903221] LNet: 111518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1942.917548] LNetError: 111518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1942.937571] LNet: Removed LNI 192.168.206.47@tcp [ 1943.537394] Key type .llcrypt unregistered [ 1943.544354] Key type ._llcrypt unregistered [ 1948.901062] Key type ._llcrypt registered [ 1948.902956] Key type .llcrypt registered [ 1949.056903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1949.066651] alg: No test for adler32 (adler32-zlib) [ 1949.964794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1949.993335] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1950.016336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1955.741233] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 22:26:30 (1788834390) [ 1957.307568] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1959.558430] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 22:26:33 (1788834393) [ 1961.190973] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1962.995447] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 22:26:37 (1788834397) [ 1963.113771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.149233] LNet: 112934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.154251] LNetError: 112934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.184962] LNet: Removed LNI 192.168.206.47@tcp [ 1963.219586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.273529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1963.298560] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1963.941415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1963.988293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1970.156917] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 22:26:45 (1788834405) [ 1970.371774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1970.404089] LNet: 113648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1970.410981] LNetError: 113648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1970.428318] LNet: Removed LNI 192.168.206.47@tcp [ 1970.454470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.516335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1970.559581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1971.219199] Key type .llcrypt unregistered [ 1971.221294] Key type ._llcrypt unregistered [ 1981.499763] Key type ._llcrypt registered [ 1981.502675] Key type .llcrypt registered [ 1982.554873] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 22:26:57 (1788834417) [ 1982.960605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1982.984198] alg: No test for adler32 (adler32-zlib) [ 1983.873760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1983.931700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1983.958165] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1984.569253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1984.631761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1990.206907] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 22:27:04 (1788834424) [ 1990.297334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1990.323977] LNet: 115196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.329695] LNetError: 115196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1990.349493] LNet: Removed LNI 192.168.206.47@tcp [ 1990.372839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1990.441396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1990.464748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1995.787549] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 22:27:10 (1788834430) [ 1995.920402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1995.961225] LNet: 115777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1995.984238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1996.050587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1996.342874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1996.381971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1996.448832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1996.782719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1996.811313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1996.911242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1996.955055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1997.023765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1997.476495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1997.501537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1997.557518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1997.978418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1998.022147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2003.614658] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 22:27:18 (1788834438) [ 2003.917390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2003.940810] LNet: 116417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.946724] LNet: 116417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2003.982208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2004.056587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2004.092931] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2004.687092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2004.706468] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 2005.348414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2005.406607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2005.592829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2005.617645] LNetError: 116696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.623135] LNet: Removed LNI 192.168.206.47@tcp [ 2006.625384] LNetError: 116696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.642783] LNet: Removed LNI 192.168.206.47@tcp2 [ 2006.680497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.753650] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2006.761895] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 2006.769473] LNet: 116703:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2007.237638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2007.429816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2007.449418] LNet: 116740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.454199] LNet: 116740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2008.479166] LNet: Removed LNI 192.168.206.47@tcp [ 2008.487906] LNet: Removed LNI 192.168.206.47@tcp2 [ 2008.505502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2008.546292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2008.562646] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2009.110564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2009.132154] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 2009.797144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2009.830884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2009.867993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2010.063990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2010.086540] LNetError: 117024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2010.093534] LNetError: 117024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2010.100496] LNet: Removed LNI 192.168.206.47@tcp [ 2011.109449] LNet: Removed LNI 192.168.206.47@tcp2 [ 2011.154718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2011.230486] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2011.236629] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 2011.246606] LNet: 117030:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2011.701306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2011.899653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2011.916375] LNet: 117068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.921294] LNet: 117068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2011.928474] LNet: Removed LNI 192.168.206.47@tcp [ 2012.965756] LNet: Removed LNI 192.168.206.47@tcp2 [ 2013.016173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.073765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2013.097713] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2013.742835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2013.761385] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 2014.401343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2014.448450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2014.656351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2014.672895] LNetError: 117347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2014.678759] LNetError: 117347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2014.685495] LNet: Removed LNI 192.168.206.47@tcp [ 2014.696876] LNet: Removed LNI 192.168.206.47@tcp2 [ 2014.722710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2014.780676] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2014.786967] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 2014.793595] LNet: 117353:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2015.201120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2015.392389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2016.415482] LNet: Removed LNI 192.168.206.47@tcp [ 2016.426185] LNet: Removed LNI 192.168.206.47@tcp2 [ 2016.457992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2016.521344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2016.544881] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2017.216424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2017.236644] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 2018.079427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2018.110990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2018.178591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2018.383566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2019.423171] LNet: Removed LNI 192.168.206.47@tcp [ 2020.456716] LNet: Removed LNI 192.168.206.47@tcp2 [ 2020.516752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2020.584019] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2020.590188] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 2020.593842] LNet: 117681:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2021.134187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2021.338961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2021.375160] LNet: 117718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2021.378660] LNet: 117718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2021.386613] LNet: Removed LNI 192.168.206.47@tcp [ 2022.437941] LNet: Removed LNI 192.168.206.47@tcp2 [ 2022.472539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.527640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2022.546438] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2023.316173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2023.343711] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 2024.132484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2024.168985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2024.236691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2024.686163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2024.709144] LNetError: 118015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2024.716448] LNetError: 118015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2024.730578] LNet: Removed LNI 192.168.206.47@tcp [ 2024.747148] LNet: Removed LNI 192.168.206.47@tcp2 [ 2024.777388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.871479] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2024.883373] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 2024.891797] LNet: 118021:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2025.717418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2026.063826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2027.168302] LNet: Removed LNI 192.168.206.47@tcp [ 2027.180552] LNet: Removed LNI 192.168.206.47@tcp2 [ 2027.208518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2027.282950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2027.314827] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2028.084670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2028.105668] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 2029.122875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2029.173331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2029.234931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2029.311369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2029.682939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2030.751389] LNet: Removed LNI 192.168.206.47@tcp [ 2031.782456] LNet: Removed LNI 192.168.206.47@tcp2 [ 2031.823419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2031.916122] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2031.928949] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 2032.605478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2039.395515] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2040.885660] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 22:27:55 (1788834475) [ 2040.966472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2041.014163] LNet: 119133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.020792] LNet: 119133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2041.032228] LNetError: 119133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.037604] LNetError: 119133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2041.052442] LNet: Removed LNI 192.168.206.47@tcp [ 2042.099177] LNet: Removed LNI 192.168.206.47@tcp2 [ 2042.168284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2042.490989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2042.521442] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2043.237588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2043.259942] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2044.156803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2044.177156] LNet: Removed LNI 10.1.2.3@tcp [ 2044.193260] LNet: Removed LNI 192.168.206.47@tcp [ 2044.219855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2044.323212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2044.351242] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2044.358079] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2045.039441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2045.079355] LNet: Removed LNI 192.168.206.47@tcp [ 2045.097469] LNet: Removed LNI 10.1.2.3@tcp [ 2052.314264] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 22:28:06 (1788834486) [ 2052.704817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2052.776375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2052.857349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2052.890480] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2053.581944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2053.632574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 2053.664876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 2053.689790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 2053.729723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 2053.782369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 2053.808205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 2053.851588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 2053.905765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 2053.938528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 2053.977798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 2054.144845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2060.714102] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 22:28:15 (1788834495) [ 2060.853865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2061.931459] LNet: Removed LNI 192.168.206.47@tcp [ 2061.992695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2062.121233] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2062.295490] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2062.298915] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2062.306194] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2064.679242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2064.708716] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2065.315243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2065.330199] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2067.128274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2067.162203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2067.251526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.47@tcp -m PUT -e local_timeout -r 1 [ 2067.336077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2067.410657] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2067.416149] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2067.699415] LNetError: 120819:0:(lib-move.c:4801:lnet_parse()) 192.168.206.47@tcp, src 192.168.206.47@tcp: Dropping PUT (error -108 looking up sender) [ 2069.471125] LNet: Removed LNI 10.1.2.3@tcp [ 2070.505660] LNet: Removed LNI 192.168.206.47@tcp [ 2070.856434] Key type .llcrypt unregistered [ 2070.858718] Key type ._llcrypt unregistered [ 2082.349901] Key type ._llcrypt registered [ 2082.357147] Key type .llcrypt registered [ 2083.492698] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 22:28:38 (1788834518) [ 2083.864608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.882883] alg: No test for adler32 (adler32-zlib) [ 2084.817739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2085.220945] LNet: 122128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.719525] Key type .llcrypt unregistered [ 2085.723756] Key type ._llcrypt unregistered [ 2092.270674] Key type ._llcrypt registered [ 2092.272925] Key type .llcrypt registered [ 2092.436254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2092.448168] alg: No test for adler32 (adler32-zlib) [ 2093.404239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2093.440135] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2101.004247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 2101.161544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2101.780207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.147@tcp [ 2102.006962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2102.057666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2102.102199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2102.147835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2102.191283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2102.228123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2102.257257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2102.308655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2102.330191] LNet: 122575:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2102.356270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2102.384842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2102.411422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2102.441518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2102.467114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2102.490849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2103.045545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2103.339233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2103.381173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2103.416808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2103.452777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2103.489874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2103.529668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2103.565928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2103.598810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2103.614858] LNet: 122692:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2103.655063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2103.691575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2103.711615] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.147@tcp [ 2103.742066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2103.778516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2103.813586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2103.855735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2104.498531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.147@tcp [ 2104.701898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2104.727517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2104.755471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2104.783480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2104.816984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2104.851528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2104.886435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2104.920087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2104.949190] LNet: 122810:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2104.966551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2105.016939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2105.056480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2105.084281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2105.106688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2105.148089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2105.823548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2106.068926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2106.099844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2106.123913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2106.159982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2106.241255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2106.296564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2106.328655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2106.378620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2106.417500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2106.464763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2106.516036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2106.571903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2106.613975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2106.646307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2106.968849] LNet: 122973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.975805] LNetError: 122973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2106.999829] LNet: Removed LNI 192.168.206.47@tcp [ 2107.315217] Key type .llcrypt unregistered [ 2107.317530] Key type ._llcrypt unregistered [ 2122.202944] Key type ._llcrypt registered [ 2122.204894] Key type .llcrypt registered [ 2123.507103] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 22:29:18 (1788834558) [ 2123.896815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2123.917561] alg: No test for adler32 (adler32-zlib) [ 2124.810346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2125.235901] LNet: 123858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.804417] Key type .llcrypt unregistered [ 2125.806596] Key type ._llcrypt unregistered [ 2131.171772] Key type ._llcrypt registered [ 2131.175502] Key type .llcrypt registered [ 2131.407885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2131.454304] alg: No test for adler32 (adler32-zlib) [ 2132.394291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2132.418383] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2139.720219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 2149.249303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.300754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2149.334737] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 2150.304963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2150.873269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.147@tcp [ 2151.153566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2151.192691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2151.251786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2151.306930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2151.342582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2151.384677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2151.449679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2151.494486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2151.512760] LNet: 124508:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2151.536879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2151.572354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2151.610600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2151.653101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2151.685234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2151.722481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2152.294734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2152.495862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2152.528040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2152.551668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2152.578555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2152.609812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2152.653691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2152.694988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2152.741167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2152.765180] LNet: 124625:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2152.786933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2152.843886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2152.933130] LNet: 2 peer NIs in recovery (showing 2): 192.168.206.147@tcp, 192.168.206.147@tcp1 [ 2152.946806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2152.997831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2153.085772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2153.135444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2153.786497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.147@tcp [ 2154.034614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2154.074552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2154.114874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2154.151902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2154.202807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2154.250823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2154.311971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2154.364299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2154.402753] LNet: 124742:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2154.433504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2154.472542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2154.519962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2154.602239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2154.653399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2154.717327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2155.381821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2155.599281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2155.636822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2155.666913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2155.693924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2155.723927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2155.753114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2155.782794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2155.813729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2155.841669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2155.880494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2155.913438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2155.939289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2155.968437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2155.992886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2160.976710] LNet: 124941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2160.987905] LNetError: 124941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2161.000803] LNet: Removed LNI 192.168.206.47@tcp [ 2161.015351] LNet: Removed LNI 192.168.206.47@tcp1 [ 2161.432911] Key type .llcrypt unregistered [ 2161.435257] Key type ._llcrypt unregistered [ 2177.578688] Key type ._llcrypt registered [ 2177.580858] Key type .llcrypt registered [ 2178.737302] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 22:30:13 (1788834613) [ 2179.051043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2179.067677] alg: No test for adler32 (adler32-zlib) [ 2179.960722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.052614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2180.077837] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2180.835390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2180.857812] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 2181.694800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2181.767556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2182.815211] LNet: Removed LNI 192.168.206.47@tcp [ 2188.107623] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 22:30:22 (1788834622) [ 2189.197564] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2190.679287] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 22:30:25 (1788834625) [ 2197.385415] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 22:30:32 (1788834632) [ 2204.588328] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 22:30:38 (1788834638) [ 2211.865438] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 22:30:46 (1788834646) [ 2218.521284] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 22:30:52 (1788834652) [ 2224.625161] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 22:30:59 (1788834659) [ 2230.260064] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 22:31:05 (1788834665) [ 2235.834966] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 22:31:10 (1788834670) [ 2241.879619] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 22:31:16 (1788834676) [ 2247.885704] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 22:31:22 (1788834682) [ 2254.037510] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 22:31:28 (1788834688) [ 2261.442513] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 22:31:35 (1788834695) [ 2267.954256] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 22:31:42 (1788834702) [ 2273.657557] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 22:31:48 (1788834708) [ 2280.863785] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 22:31:55 (1788834715) [ 2289.387246] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 22:32:03 (1788834723) [ 2296.996069] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 22:32:11 (1788834731) [ 2304.094489] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 22:32:18 (1788834738) [ 2304.267966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2304.285176] LNet: 136633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.291152] LNetError: 136633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2304.303321] LNet: Removed LNI 192.168.206.47@tcp2 [ 2304.323773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.461569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2304.481884] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2305.667506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2305.706417] LNet: 136775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.718222] LNetError: 136775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2305.748149] LNet: Removed LNI 192.168.206.47@tcp [ 2305.767602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2305.874162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2305.918030] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2312.951211] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 22:32:27 (1788834747) [ 2321.147212] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 22:32:35 (1788834755) [ 2327.864732] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 22:32:42 (1788834762) [ 2334.153873] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 22:32:48 (1788834768) [ 2341.590344] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 22:32:56 (1788834776) [ 2349.510898] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 22:33:03 (1788834783) [ 2356.663901] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 22:33:11 (1788834791) [ 2364.413995] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 22:33:19 (1788834799) [ 2371.516421] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 22:33:26 (1788834806) [ 2378.663644] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 22:33:33 (1788834813) [ 2385.078388] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 22:33:39 (1788834819) [ 2391.362455] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 22:33:46 (1788834826) [ 2397.363430] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2114 sec ========== 22:33:52 (1788834832) [ 2397.586904] LNet: 144240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2397.599105] LNetError: 144240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2397.617192] LNet: Removed LNI 192.168.206.47@tcp [ 2398.194668] Key type .llcrypt unregistered [ 2398.199981] Key type ._llcrypt unregistered [ 2408.815323] Key type ._llcrypt registered [ 2408.817851] Key type .llcrypt registered [ 2409.218376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2409.240034] alg: No test for adler32 (adler32-zlib) [ 2410.339609] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 2410.571970] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2412.223547] Key type lgssc registered [ 2413.586534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2473.567692] Lustre: Mounted lustre-client - version 2.17.57_110_g063755c [ 2479.355674] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2486.986848] Lustre: Unmounted lustre-client [ 2503.366913] Key type lgssc unregistered [ 2503.660805] LNet: 147185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2503.671644] LNetError: 147185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2503.689284] LNet: Removed LNI 192.168.206.47@tcp [ 2504.382158] Key type .llcrypt unregistered [ 2504.385890] Key type ._llcrypt unregistered