[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 466742203 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002426] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007334] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011173] LSM: Security Framework initializing [ 0.012065] Yama: becoming mindful. [ 0.013035] SELinux: Initializing. [ 0.014105] *** VALIDATE selinux *** [ 0.025050] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030605] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031200] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032132] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034024] *** VALIDATE tmpfs *** [ 0.036078] *** VALIDATE proc *** [ 0.037273] *** VALIDATE cgroup *** [ 0.038012] *** VALIDATE cgroup2 *** [ 0.039295] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042021] Spectre V2 : User space: Vulnerable [ 0.043006] Speculative Store Bypass: Vulnerable [ 0.046693] debug: unmapping init [mem 0xffffffff9c859000-0xffffffff9c860fff] [ 0.049202] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050804] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051028] ... version: 2 [ 0.052021] ... bit width: 48 [ 0.053020] ... generic registers: 4 [ 0.054017] ... value mask: 0000ffffffffffff [ 0.055022] ... max period: 00007fffffffffff [ 0.056016] ... fixed-purpose events: 3 [ 0.057013] ... event mask: 000000070000000f [ 0.058323] rcu: Hierarchical SRCU implementation. [ 0.060495] smp: Bringing up secondary CPUs ... [ 0.061573] x86: Booting SMP configuration: [ 0.062032] .... node #0, CPUs: #1 #2 #3 [ 0.065459] smp: Brought up 1 node, 4 CPUs [ 0.067016] smpboot: Max logical packages: 1 [ 0.068019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.152245] node 0 deferred pages initialised in 82ms [ 0.155261] devtmpfs: initialized [ 0.156250] x86/mm: Memory block size: 128MB [ 0.159736] gcov: version magic: 0x41383552 [ 0.163181] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.164109] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165281] pinctrl core: initialized pinctrl subsystem [ 0.166157] [ 0.166556] ************************************************************* [ 0.167014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168012] ** ** [ 0.169024] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170011] ** ** [ 0.171011] ** This means that this kernel is built to expose internal ** [ 0.172012] ** IOMMU data structures, which may compromise security on ** [ 0.173016] ** your system. ** [ 0.174013] ** ** [ 0.175012] ** If you see this message and you are not debugging the ** [ 0.176018] ** kernel, report this immediately to your vendor! ** [ 0.177011] ** ** [ 0.178014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179016] ************************************************************* [ 0.180606] NET: Registered protocol family 16 [ 0.181400] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.183065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.184404] cpuidle: using governor menu [ 0.186814] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.188413] PCI: Using configuration type 1 for base access [ 0.190127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.199124] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.200033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.202019] cryptd: max_cpu_qlen set to 1000 [ 0.204167] ACPI: Added _OSI(Module Device) [ 0.205012] ACPI: Added _OSI(Processor Device) [ 0.205940] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.210007] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.212548] ACPI: Interpreter enabled [ 0.213068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214012] ACPI: Using IOAPIC for interrupt routing [ 0.215105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.216409] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.226239] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.227040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.228065] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.229091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.231164] acpiphp: Slot [2] registered [ 0.232143] acpiphp: Slot [5] registered [ 0.233097] acpiphp: Slot [6] registered [ 0.234102] acpiphp: Slot [3] registered [ 0.235088] acpiphp: Slot [4] registered [ 0.236097] acpiphp: Slot [7] registered [ 0.237126] acpiphp: Slot [8] registered [ 0.238124] acpiphp: Slot [9] registered [ 0.239094] acpiphp: Slot [10] registered [ 0.240093] acpiphp: Slot [11] registered [ 0.241133] acpiphp: Slot [12] registered [ 0.242127] acpiphp: Slot [13] registered [ 0.243150] acpiphp: Slot [14] registered [ 0.244099] acpiphp: Slot [15] registered [ 0.245106] acpiphp: Slot [16] registered [ 0.246117] acpiphp: Slot [17] registered [ 0.247111] acpiphp: Slot [18] registered [ 0.248123] acpiphp: Slot [19] registered [ 0.249121] acpiphp: Slot [20] registered [ 0.250096] acpiphp: Slot [21] registered [ 0.251085] acpiphp: Slot [22] registered [ 0.252098] acpiphp: Slot [23] registered [ 0.253081] acpiphp: Slot [24] registered [ 0.254161] acpiphp: Slot [25] registered [ 0.255123] acpiphp: Slot [26] registered [ 0.256217] acpiphp: Slot [27] registered [ 0.257115] acpiphp: Slot [28] registered [ 0.258121] acpiphp: Slot [29] registered [ 0.259126] acpiphp: Slot [30] registered [ 0.260119] acpiphp: Slot [31] registered [ 0.261069] PCI host bridge to bus 0000:00 [ 0.262025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.263026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.264034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.267027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.271028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.276032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.279220] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.284042] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.288360] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.298014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.303061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.307021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.311021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.314020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.317399] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.322242] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.325045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.330239] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.335066] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.346025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.352016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.357285] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.365016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.377016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.390017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.399964] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.408022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.413014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.431028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.451490] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.454422] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.456566] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.458357] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.461235] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.465150] iommu: Default domain type: Passthrough [ 0.468224] SCSI subsystem initialized [ 0.469113] ACPI: bus type USB registered [ 0.471105] usbcore: registered new interface driver usbfs [ 0.472099] usbcore: registered new interface driver hub [ 0.474120] usbcore: registered new device driver usb [ 0.475199] pps_core: LinuxPPS API ver. 1 registered [ 0.478079] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.480061] PTP clock support registered [ 0.482065] EDAC MC: Ver: 3.0.0 [ 0.484064] PCI: Using ACPI for IRQ routing [ 0.485490] NetLabel: Initializing [ 0.486014] NetLabel: domain hash size = 128 [ 0.487015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.489088] NetLabel: unlabeled traffic allowed by default [ 0.492054] vgaarb: loaded [ 0.493398] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.498065] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.505000] clocksource: Switched to clocksource kvm-clock [ 0.610030] VFS: Disk quotas dquot_6.6.0 [ 0.611775] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.614340] *** VALIDATE ramfs *** [ 0.616304] *** VALIDATE hugetlbfs *** [ 0.618326] pnp: PnP ACPI init [ 0.620546] pnp: PnP ACPI: found 6 devices [ 0.639650] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.643086] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.645565] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.647614] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.649972] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.652642] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.655656] NET: Registered protocol family 2 [ 0.658396] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.663273] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.667202] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.674304] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.677685] TCP: Hash tables configured (established 65536 bind 65536) [ 0.680849] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.684509] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.687734] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.690535] NET: Registered protocol family 1 [ 0.693286] RPC: Registered named UNIX socket transport module. [ 0.697574] RPC: Registered udp transport module. [ 0.699323] RPC: Registered tcp transport module. [ 0.701072] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.703195] NET: Registered protocol family 44 [ 0.705015] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.707373] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.709553] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.712079] PCI: CLS 0 bytes, default 64 [ 0.713749] Unpacking initramfs... [ 2.133270] debug: unmapping init [mem 0xffff9e31bcc64000-0xffff9e31bffcffff] [ 2.137247] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.139615] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.142536] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.647048] Initialise system trusted keyrings [ 2.648381] Key type blacklist registered [ 2.651568] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.660296] zbud: loaded [ 2.662830] *** VALIDATE nfs *** [ 2.663829] *** VALIDATE nfs4 *** [ 2.665266] pstore: using deflate compression [ 2.672253] Platform Keyring initialized [ 2.797116] NET: Registered protocol family 38 [ 2.798739] Key type asymmetric registered [ 2.800573] Asymmetric key parser 'x509' registered [ 2.801870] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.804206] io scheduler mq-deadline registered [ 2.805474] io scheduler kyber registered [ 2.806686] io scheduler bfq registered [ 2.808637] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.812513] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.815924] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.818961] ACPI: Power Button [PWRF] [ 2.826201] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.834291] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.845435] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.874835] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.902132] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.907806] Non-volatile memory driver v1.3 [ 2.910216] Linux agpgart interface v0.103 [ 2.938446] virtio_blk virtio1: [vda] 149984 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.940712] vda: detected capacity change from 0 to 76791808 [ 2.954250] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.956918] vdb: detected capacity change from 0 to 1073741824 [ 2.962650] libphy: Fixed MDIO Bus: probed [ 2.968564] usbcore: registered new interface driver usbserial_generic [ 2.971565] usbserial: USB Serial support registered for generic [ 2.974463] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.979145] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.980990] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.983777] mousedev: PS/2 mouse device common for all mice [ 2.986983] rtc_cmos 00:05: RTC can wake from S4 [ 2.990499] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.990571] rtc_cmos 00:05: registered as rtc0 [ 2.996277] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.998724] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.999682] intel_pstate: CPU model not supported [ 3.004734] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.009279] hid: raw HID events driver (C) Jiri Kosina [ 3.011421] usbcore: registered new interface driver usbhid [ 3.013096] usbhid: USB HID core driver [ 3.014904] drop_monitor: Initializing network drop monitor service [ 3.017406] Initializing XFRM netlink socket [ 3.019711] NET: Registered protocol family 10 [ 3.022410] Segment Routing with IPv6 [ 3.023921] NET: Registered protocol family 17 [ 3.025803] mpls_gso: MPLS GSO support [ 3.031437] RAS: Correctable Errors collector initialized. [ 3.033956] AVX version of gcm_enc/dec engaged. [ 3.035751] AES CTR mode by8 optimization enabled [ 3.108524] sched_clock: Marking stable (3108491325, 0)->(4082110721, -973619396) [ 3.112612] registered taskstats version 1 [ 3.114938] Loading compiled-in X.509 certificates [ 3.117143] zswap: loaded using pool lzo/zbud [ 3.149246] Key type big_key registered [ 3.163729] Key type encrypted registered [ 3.165447] ima: No TPM chip found, activating TPM-bypass! [ 3.167376] ima: Allocated hash algorithm: sha1 [ 3.169188] ima: No architecture policies found [ 3.171092] evm: Initialising EVM extended attributes: [ 3.173081] evm: security.selinux [ 3.174341] evm: security.ima [ 3.175540] evm: security.capability [ 3.176970] evm: HMAC attrs: 0x1 [ 3.179645] rtc_cmos 00:05: setting system clock to 2026-09-09 16:01:34 UTC (1788969694) [ 3.185546] debug: unmapping init [mem 0xffffffff9d803000-0xffffffff9d9fffff] [ 3.188817] debug: unmapping init [mem 0xffffffff9c582000-0xffffffff9c858fff] [ 3.198216] Write protecting the kernel read-only data: 28672k [ 3.202344] debug: unmapping init [mem 0xffffffff9ac03000-0xffffffff9adfffff] [ 3.205559] debug: unmapping init [mem 0xffffffff9b514000-0xffffffff9b5fffff] [ 3.238118] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.248404] systemd[1]: Detected virtualization kvm. [ 3.250154] systemd[1]: Detected architecture x86-64. [ 3.252527] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.275106] systemd[1]: No hostname configured. [ 3.276844] systemd[1]: Set hostname to . [ 3.278930] random: systemd: uninitialized urandom read (16 bytes read) [ 3.281063] systemd[1]: Initializing machine ID from random generator. [ 3.316990] random: ln: uninitialized urandom read (6 bytes read) [ 3.406935] random: systemd: uninitialized urandom read (16 bytes read) [ 3.409549] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.414205] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.418130] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.002314] device-mapper: uevent: version 1.0.3 [ 4.004596] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.825853] virtio_net virtio0 ens2: renamed from eth0 [ 4.923510] scsi host0: ata_piix [ 4.953241] scsi host1: ata_piix [ 4.954568] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.956533] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 7.010571] random: fast init done [ 8.808812] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.683616] random: crng init done [ 9.685181] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.238785] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.052158] hrtimer: interrupt took 2270851 ns [ 13.324611] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.374783] SELinux: Disabled at runtime. [ 14.566870] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.601655] systemd[1]: Detected virtualization kvm. [ 14.608397] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.717395] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.745301] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.764315] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.775769] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.792979] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.845854] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.864719] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd File Systems. [ 17.053644] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 18.503917] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.301769] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.319454] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.706142] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.815842] EDAC sbridge: Ver: 1.1.2 [ 23.312614] Key type dns_resolver registered [ 24.033797] NFS: Registering the id_resolver key type [ 24.043167] Key type id_resolver registered [ 24.048801] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg402-client login: [ 85.803253] libcfs: loading out-of-tree module taints kernel. [ 86.154783] Key type ._llcrypt registered [ 86.176523] Key type .llcrypt registered [ 86.932982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 86.950160] alg: No test for adler32 (adler32-zlib) [ 88.663853] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 90.644874] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 92.969119] Key type lgssc registered [ 96.928953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 227.626161] Lustre: Mounted lustre-client - version 2.17.58_42_gf11d80b [ 232.730403] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 248.390384] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing check_logdir /tmp/testlogs/ [ 253.407576] Lustre: lustre-OST0000-osc-ffff9e32056d7800: disconnect after 23s idle [ 254.156358] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing yml_node [ 258.488166] Lustre: DEBUG MARKER: Client: 2.17.58.42 [ 261.517182] Lustre: DEBUG MARKER: MDS: 2.17.58.42 [ 264.115740] Lustre: DEBUG MARKER: OSS: 2.17.58.42 [ 266.086359] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 9 12:05:55 EDT 2026 [ 283.320096] Lustre: Unmounted lustre-client [ 308.674392] Key type lgssc unregistered [ 309.142757] LNet: 8413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.149340] LNetError: 8413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 309.170252] LNet: Removed LNI 192.168.204.2@tcp [ 309.988385] Key type .llcrypt unregistered [ 309.993068] Key type ._llcrypt unregistered [ 335.885294] Key type ._llcrypt registered [ 335.890310] Key type .llcrypt registered [ 336.240493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 336.257964] alg: No test for adler32 (adler32-zlib) [ 337.179709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 337.251250] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 337.269947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 340.946928] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 341.042457] LNet: 9285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.055794] LNetError: 9285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 341.079974] LNet: Removed LNI 192.168.204.2@tcp [ 341.718455] Key type .llcrypt unregistered [ 341.723338] Key type ._llcrypt unregistered [ 347.298691] Key type ._llcrypt registered [ 347.300717] Key type .llcrypt registered [ 348.388472] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:07:18 (1788970038) [ 348.729593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 348.755646] alg: No test for adler32 (adler32-zlib) [ 349.705433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.813476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.836925] LNet: 9702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.867936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.935703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 349.959550] LNet: 9709:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 356.357927] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:07:26 (1788970046) [ 356.528103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.572752] LNet: 10286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.597585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.706878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 362.795239] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:07:32 (1788970052) [ 362.927581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.993627] LNet: 10865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.027749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.173943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 370.078925] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:07:40 (1788970060) [ 370.282234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.299214] LNet: 11459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.320480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.418925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 377.237862] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:07:47 (1788970067) [ 377.310332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.344101] LNet: 12039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.373560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.457561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 377.849063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 384.726732] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:07:54 (1788970074) [ 384.882709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.992182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.087604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 385.771408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 386.570505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 394.538461] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:08:04 (1788970084) [ 394.868814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.895751] LNet: 13230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.905713] LNet: 13230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 394.948141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.052495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 403.198330] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:08:12 (1788970092) [ 403.350377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.442995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.592187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 403.676160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 404.512139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 404.564368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 405.300379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 405.368509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 406.046129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 406.120836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 406.661323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 406.691629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 407.290492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 407.363851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 407.957868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 407.982588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 408.582863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 408.623743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 409.212868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 409.270421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 409.916655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 410.010040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 410.478885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 410.505924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 411.147877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 411.215570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 411.851108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 411.899940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 419.895436] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:08:29 (1788970109) [ 420.044653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.083138] LNet: 14495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.086466] LNet: 14495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 420.127757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.237943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 420.304828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 428.883806] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:08:38 (1788970118) [ 429.193492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.249946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.399422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 429.456707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 437.491220] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:08:47 (1788970127) [ 437.655305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.726580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.930203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 437.966142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 445.652990] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:08:55 (1788970135) [ 445.774480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.830567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.935892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 446.000954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 453.340872] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:09:03 (1788970143) [ 453.542817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.594094] LNet: 16843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.611942] LNet: 16843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 453.666326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.763772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 453.802871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 460.562567] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:09:10 (1788970150) [ 460.729980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.785239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.877961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 460.916427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 466.902466] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:09:16 (1788970156) [ 467.022243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.071243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.182780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 467.613967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 475.034916] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:09:24 (1788970164) [ 475.192895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.253310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.493278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 475.723169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 482.588657] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:09:32 (1788970172) [ 482.948995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.980760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.100306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 483.410133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 489.642906] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:09:39 (1788970179) [ 489.811504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.892190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.416584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 490.636753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 497.618756] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:09:47 (1788970187) [ 497.936287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.997985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.082957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 498.124264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 505.011313] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:09:54 (1788970194) [ 505.153407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.230566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.395936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 505.440955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 512.965746] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:10:02 (1788970202) [ 513.207329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.281695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.388863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 520.029892] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:10:10 (1788970210) [ 520.137820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.154658] LNet: 22282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 520.163209] LNet: 22282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 520.216395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.333230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 525.993472] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:10:16 (1788970216) [ 526.043802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.079145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.211546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 526.624564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 535.126468] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:10:25 (1788970225) [ 535.336787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.368149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.406340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 535.431852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 542.640746] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:10:32 (1788970232) [ 542.810926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.894901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.048532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 543.103942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 551.660993] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:10:41 (1788970241) [ 552.094592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.143820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.256399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 552.323427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 558.622792] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:10:48 (1788970248) [ 558.824639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.872281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.979792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 559.028893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 565.932348] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:10:55 (1788970255) [ 566.014537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.086355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.169909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 566.274157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 566.291236] LNetError: 25818:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 566.323837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 566.364184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 571.895523] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:11:01 (1788970261) [ 572.061382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.166603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.289701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.viEc4jXKN1 [ 572.374640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.viEc4jXKN1 [ 572.453328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.viEc4jXKN1 [ 578.449307] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:11:08 (1788970268) [ 578.876885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.954626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.027546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 579.079662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 579.154691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 586.872966] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 12:11:16 (1788970276) [ 586.991869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.051641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.205686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 587.253831] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 588.916468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 590.478353] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 592.559408] Key type lgssc registered [ 594.099036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 599.202818] LNetError: 27578:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.2@tcp99 has no available nets [ 602.504728] Key type lgssc unregistered [ 602.831962] LNetError: 28687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 603.878357] LNet: Removed LNI 192.168.204.2@tcp [ 604.700365] Key type .llcrypt unregistered [ 604.704602] Key type ._llcrypt unregistered [ 611.831824] Key type ._llcrypt registered [ 611.835084] Key type .llcrypt registered [ 612.943354] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:11:43 (1788970303) [ 613.802630] Key type .llcrypt unregistered [ 613.810909] Key type ._llcrypt unregistered [ 620.385576] Key type ._llcrypt registered [ 620.396104] Key type .llcrypt registered [ 620.638741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 620.653463] alg: No test for adler32 (adler32-zlib) [ 621.614964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.701665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 621.721512] LNet: 29597:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 621.728933] LNet: 29597:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 621.738201] LNet: 29597:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 621.750278] LNet: Message forwarding will be enabled in 2s [ 624.864405] LNet: Message forwarding enabled [ 625.350701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 625.386840] LNet: Message forwarding will stop in 2s to 6s [ 628.969637] LNet: Message forwarding disabled [ 629.963271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 629.994237] LNet: 29695:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 630.004817] LNet: 29695:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 630.011846] LNet: 29695:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 630.024440] LNet: Message forwarding will be enabled in 2s [ 630.243201] LNet: 29708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 631.166649] Key type .llcrypt unregistered [ 631.172816] Key type ._llcrypt unregistered [ 644.677502] Key type ._llcrypt registered [ 644.679322] Key type .llcrypt registered [ 645.991575] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:12:15 (1788970335) [ 646.438707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 646.471444] alg: No test for adler32 (adler32-zlib) [ 647.412707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.505324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 654.122709] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:12:23 (1788970343) [ 654.252333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.319721] LNet: 31104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.355704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.423283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 654.448682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 654.479565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 654.507432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 654.532612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 654.564422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 654.587388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 654.616931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 654.654878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 654.716670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 654.765337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 654.811721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 654.849447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 654.920811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 654.999748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 655.077460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 655.145433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 655.189760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 655.229652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 655.257855] LNetError: 31152:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 662.304394] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:12:32 (1788970352) [ 662.446975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.475588] LNet: 31718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.511535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.667231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 669.778532] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:12:39 (1788970359) [ 671.708225] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg402-server [ 673.556473] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:12:43 (1788970363) [ 673.673785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.715748] LNet: 32649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.748612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.823966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 673.882359] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 674.871598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[3-7]@tcp [ 674.908035] LNet: 32790:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 674.979386] LNetError: 32657:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.3@tcp is being used as a gateway but routing feature is not turned on [ 674.993935] LNetError: 32657:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.3@tcp has gone from up to down [ 682.248857] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:12:52 (1788970372) [ 684.425231] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg402-server [ 686.500411] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:12:56 (1788970376) [ 686.798597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.824908] LNet: 33726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.830629] LNetError: 33726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.875219] LNet: Removed LNI 192.168.204.2@tcp [ 686.933532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.981668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 687.004648] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 687.885747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[3-8/2]@tcp [ 687.935471] LNet: 33843:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 687.948501] LNet: 33843:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 687.977995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[3-8/2]@tcp [ 688.000564] LNetError: 33730:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.5@tcp is being used as a gateway but routing feature is not turned on [ 688.026851] LNetError: 33730:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 688.043240] LNetError: 33730:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.5@tcp has gone from up to down [ 688.064087] LNetError: 33730:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 695.075309] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:13:04 (1788970384) [ 695.342628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.378038] LNet: 34436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.392948] LNetError: 34436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 695.416416] LNet: Removed LNI 192.168.204.2@tcp [ 695.485378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.593207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 695.692549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.761599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.859678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 695.953437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.998366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.057052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 696.145826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.213522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.323938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.451555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.524566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.578238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 696.736223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.790609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.891273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.951591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.976610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.070230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 697.162469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.202711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.261148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.332947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.365653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.408821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 697.478270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.541627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.637594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.769428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.811700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.875243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 697.930253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.975380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.050181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 704.763608] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:13:14 (1788970394) [ 706.577445] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg402-server [ 708.187178] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:13:18 (1788970398) [ 710.161956] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg402-server [ 712.128800] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:13:21 (1788970401) [ 712.434987] LNet: 35838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.445183] LNet: 35838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 713.343872] Key type .llcrypt unregistered [ 713.349285] Key type ._llcrypt unregistered [ 721.237731] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 721.361549] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 721.367237] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 721.372161] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 723.939596] Key type ._llcrypt registered [ 723.943811] Key type .llcrypt registered [ 724.163301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 724.179138] alg: No test for adler32 (adler32-zlib) [ 725.102159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.143623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 725.161709] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 725.713177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 725.738762] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 727.923581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 727.945670] LNet: Removed LNI 10.1.2.3@tcp [ 728.086913] LNet: 36557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.093132] LNetError: 36557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.105933] LNet: Removed LNI 192.168.204.2@tcp [ 728.819422] Key type .llcrypt unregistered [ 728.824749] Key type ._llcrypt unregistered [ 740.277695] Key type ._llcrypt registered [ 740.280953] Key type .llcrypt registered [ 741.652105] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:13:51 (1788970431) [ 742.233419] Key type .llcrypt unregistered [ 742.237172] Key type ._llcrypt unregistered [ 748.089778] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 748.343493] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 748.352669] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 748.380429] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 751.136808] Key type ._llcrypt registered [ 751.138787] Key type .llcrypt registered [ 751.371787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 751.406206] alg: No test for adler32 (adler32-zlib) [ 752.390900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.473093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 752.517565] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 753.408855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 753.480961] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 757.083279] LNet: 38053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.090981] LNetError: 38053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 757.099123] LNet: Removed LNI 10.1.2.3@tcp [ 757.112680] LNet: Removed LNI 192.168.204.2@tcp [ 757.909188] Key type .llcrypt unregistered [ 757.910742] Key type ._llcrypt unregistered [ 772.379861] Key type ._llcrypt registered [ 772.381574] Key type .llcrypt registered [ 773.771549] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:14:23 (1788970463) [ 774.602320] Key type .llcrypt unregistered [ 774.605143] Key type ._llcrypt unregistered [ 781.825167] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 781.971927] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 781.975882] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 781.980330] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 784.518810] Key type ._llcrypt registered [ 784.522455] Key type .llcrypt registered [ 784.768425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 784.779968] alg: No test for adler32 (adler32-zlib) [ 785.668940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.761965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 785.778213] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 787.915769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 787.940693] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 790.795526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 790.842229] LNet: Removed LNI 10.1.2.3@tcp [ 790.863405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 790.900503] LNet: Removed LNI 10.1.2.31@tcp [ 790.921544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 790.997277] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 792.115530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 792.164115] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 793.564440] LNet: 39967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.570397] LNetError: 39967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 793.579884] LNet: Removed LNI 10.1.2.31@tcp [ 793.598509] LNet: Removed LNI 10.1.2.3@tcp [ 794.641570] Key type .llcrypt unregistered [ 794.650502] Key type ._llcrypt unregistered [ 808.276235] Key type ._llcrypt registered [ 808.281078] Key type .llcrypt registered [ 809.721405] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:14:59 (1788970499) [ 810.063365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 810.081709] alg: No test for adler32 (adler32-zlib) [ 811.022259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.111329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.157312] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 812.168979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.193216] LNet: 40917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.197535] LNetError: 40917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.218840] LNet: Removed LNI 192.168.204.2@tcp [ 812.275509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.394304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.2@tcp [ 812.415108] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 813.322394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.2@tcp [ 813.359213] LNet: Removed LNI 192.168.204.2@tcp [ 813.442386] LNet: 41052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.406185] Key type .llcrypt unregistered [ 814.410803] Key type ._llcrypt unregistered [ 829.037658] Key type ._llcrypt registered [ 829.040685] Key type .llcrypt registered [ 830.485741] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:15:20 (1788970520) [ 832.265315] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg402-server [ 834.170348] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:15:23 (1788970523) [ 834.784333] Key type .llcrypt unregistered [ 834.790412] Key type ._llcrypt unregistered [ 840.437533] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 840.764565] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 840.768927] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 840.775068] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 843.524248] Key type ._llcrypt registered [ 843.527673] Key type .llcrypt registered [ 843.855714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 843.898574] alg: No test for adler32 (adler32-zlib) [ 844.888719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.066933] LNet: Added LNI 192.168.204.2@tcp1 [8/256/0/180] [ 846.333597] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 848.670778] LNet: 42884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.680785] LNetError: 42884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.690681] LNet: Removed LNI 192.168.204.2@tcp1 [ 848.702552] LNet: Removed LNI 10.1.2.3@tcp2 [ 849.354420] Key type .llcrypt unregistered [ 849.357453] Key type ._llcrypt unregistered [ 862.654918] Key type ._llcrypt registered [ 862.658914] Key type .llcrypt registered [ 863.986366] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:15:53 (1788970553) [ 864.643855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 864.670654] alg: No test for adler32 (adler32-zlib) [ 865.722604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.818491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 865.885974] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 867.056883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 867.090943] LNet: 43852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.098044] LNetError: 43852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.126936] LNet: Removed LNI 192.168.204.2@tcp [ 867.140815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.207451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 867.234253] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 875.845943] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:16:05 (1788970565) [ 876.105532] LNet: 44440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.116854] LNetError: 44440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.161905] LNet: Removed LNI 192.168.204.2@tcp [ 876.947680] Key type .llcrypt unregistered [ 876.953097] Key type ._llcrypt unregistered [ 884.707286] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 885.138231] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 885.144052] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 885.148660] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 887.759970] Key type ._llcrypt registered [ 887.763955] Key type .llcrypt registered [ 887.965159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 887.977692] alg: No test for adler32 (adler32-zlib) [ 888.858533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.914799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 888.968525] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 890.026173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 890.043442] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 892.445545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.467299] LNet: 45183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.472558] LNetError: 45183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 892.477635] LNet: Removed LNI 10.1.2.3@tcp [ 892.487698] LNet: Removed LNI 192.168.204.2@tcp [ 892.504493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.558849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 892.579107] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 892.583676] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 892.944940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.968771] LNet: Removed LNI 10.1.2.3@tcp [ 892.982706] LNetError: 45210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.993847] LNetError: 45210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 893.004539] LNet: Removed LNI 192.168.204.2@tcp [ 893.018295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.065323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 893.098553] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 893.104311] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 893.551337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 893.571291] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 894.613692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.661706] LNet: 45372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.675773] LNet: 45372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 894.687244] LNetError: 45372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 894.694398] LNet: Removed LNI 10.1.2.3@tcp [ 894.696364] LNet: Removed LNI 192.168.204.2@tcp [ 894.706199] LNet: Removed LNI 192.168.204.2@tcp2 [ 894.730103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.786571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 894.806817] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 894.811098] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 894.815911] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 902.101916] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:16:31 (1788970591) [ 902.323809] LNet: 45966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.342536] LNetError: 45966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 902.354412] LNetError: 45966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 902.363085] LNet: Removed LNI 10.1.2.3@tcp [ 902.369755] LNet: Removed LNI 192.168.204.2@tcp [ 902.382354] LNet: Removed LNI 192.168.204.2@tcp2 [ 903.310035] Key type .llcrypt unregistered [ 903.322318] Key type ._llcrypt unregistered [ 909.852568] Key type ._llcrypt registered [ 909.854874] Key type .llcrypt registered [ 910.084941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 910.107620] alg: No test for adler32 (adler32-zlib) [ 911.270976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.354661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 911.380933] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 912.383806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.395539] LNet: 46363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.401995] LNetError: 46363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.412789] LNet: Removed LNI 192.168.204.2@tcp [ 912.431409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.478912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 912.502586] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 912.838580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.861517] LNet: Removed LNI 192.168.204.2@tcp [ 912.904289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.945625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 912.973570] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 913.282292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.296733] LNet: 46424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.300704] LNet: 46424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 913.305907] LNetError: 46424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.310524] LNetError: 46424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 913.319794] LNet: Removed LNI 192.168.204.2@tcp [ 913.368147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.412524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 913.431932] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 913.932868] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 914.153042] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 914.162984] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 914.186378] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 916.669955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 916.684299] LNet: 46502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.690639] LNetError: 46502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.705457] LNet: Removed LNI 192.168.204.2@tcp [ 916.729398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.811465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 916.828783] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 923.477839] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:16:53 (1788970613) [ 923.581963] LNet: 47092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.598988] LNetError: 47092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.617560] LNet: Removed LNI 192.168.204.2@tcp [ 924.461873] Key type .llcrypt unregistered [ 924.465233] Key type ._llcrypt unregistered [ 931.321877] Key type ._llcrypt registered [ 931.323520] Key type .llcrypt registered [ 931.535494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 931.556605] alg: No test for adler32 (adler32-zlib) [ 932.501724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.574208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 932.616654] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 933.488970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.506603] LNet: 47496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.514298] LNetError: 47496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.530519] LNet: Removed LNI 192.168.204.2@tcp [ 933.554583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.647544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 933.680454] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 940.700435] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:17:10 (1788970630) [ 940.939302] LNet: 48088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.946928] LNetError: 48088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.963433] LNet: Removed LNI 192.168.204.2@tcp [ 941.573914] Key type .llcrypt unregistered [ 941.580643] Key type ._llcrypt unregistered [ 948.749706] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 948.921379] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 948.924878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 948.945370] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 951.616186] Key type ._llcrypt registered [ 951.618947] Key type .llcrypt registered [ 951.898580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 951.937283] alg: No test for adler32 (adler32-zlib) [ 952.941688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.022716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 953.068430] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 954.529914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.562503] LNet: 48535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.576750] LNetError: 48535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.601394] LNet: Removed LNI 192.168.204.2@tcp [ 954.639584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.776197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 954.845986] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 962.347565] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:17:32 (1788970652) [ 962.629549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.652836] LNet: 49134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.659410] LNetError: 49134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.671420] LNet: Removed LNI 192.168.204.2@tcp [ 962.698249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.821668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 968.397298] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:17:38 (1788970658) [ 968.547776] LNet: 49709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.191606] Key type .llcrypt unregistered [ 969.194089] Key type ._llcrypt unregistered [ 975.046753] Key type ._llcrypt registered [ 975.049167] Key type .llcrypt registered [ 975.211070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 975.219235] alg: No test for adler32 (adler32-zlib) [ 976.142433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.287771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 976.335310] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 977.280906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 977.300988] LNet: 50111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.306498] LNetError: 50111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.320801] LNet: Removed LNI 192.168.204.2@tcp [ 977.356640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.399415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 977.422782] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 984.443693] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:17:54 (1788970674) [ 984.629824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.667348] LNet: 50701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.683808] LNetError: 50701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.723448] LNet: Removed LNI 192.168.204.2@tcp [ 984.792214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.881124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 991.389429] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:18:01 (1788970681) [ 992.939889] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 994.772186] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:18:04 (1788970684) [ 996.306870] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 998.413893] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:18:07 (1788970687) [ 1000.028657] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1001.815889] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:18:11 (1788970691) [ 1003.524444] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1005.512471] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:18:15 (1788970695) [ 1007.448979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1009.589105] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:18:19 (1788970699) [ 1009.731343] LNet: 52720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.617972] Key type .llcrypt unregistered [ 1010.620430] Key type ._llcrypt unregistered [ 1017.437880] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1017.741960] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1017.753808] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1017.767762] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1020.407533] Key type ._llcrypt registered [ 1020.410752] Key type .llcrypt registered [ 1020.651621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1020.675916] alg: No test for adler32 (adler32-zlib) [ 1021.556031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1021.657701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1021.694526] LNet: Added LNI 192.168.204.2@tcp [8/512/0/180] [ 1021.709298] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1021.711267] LNet: 53045:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1021.859943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1021.880454] LNet: 53064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.890918] LNetError: 53064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1021.895202] LNet: Removed LNI 10.1.2.3@tcp [ 1021.903232] LNetError: 53064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.924783] LNet: Removed LNI 192.168.204.2@tcp [ 1021.938199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1022.028478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1022.050387] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1022.117216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1022.145151] LNet: 53088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.158054] LNetError: 53088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.177825] LNet: Removed LNI 192.168.204.2@tcp [ 1022.209305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1022.280363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1022.299839] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1022.308730] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1022.313862] LNet: 53096:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1022.402345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1022.422437] LNet: 53114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.428588] LNetError: 53114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1022.433332] LNet: Removed LNI 10.1.2.3@tcp [ 1022.435377] LNetError: 53114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.449302] LNet: Removed LNI 192.168.204.2@tcp [ 1022.483395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1022.580325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1022.614857] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1022.621806] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1022.625650] LNet: 53122:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1022.878995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1022.905653] LNet: 53148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.912902] LNetError: 53148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1022.918638] LNet: Removed LNI 10.1.2.3@tcp [ 1022.921567] LNetError: 53148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.931221] LNet: Removed LNI 192.168.204.2@tcp [ 1022.952243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.032115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1023.065095] LNet: Added LNI 192.168.204.2@tcp [8/256/0/182] [ 1023.071233] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1023.203286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1023.227556] LNet: 53176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.233690] LNetError: 53176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1023.245921] LNet: Removed LNI 10.1.2.3@tcp [ 1023.247562] LNetError: 53176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.260882] LNet: Removed LNI 192.168.204.2@tcp [ 1023.289958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.399548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1023.424833] LNet: Added LNI 192.168.204.2@tcp [8/256/0/182] [ 1023.431140] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1023.437401] LNet: 53184:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1023.586642] LNet: Removed LNI 10.1.2.3@tcp [ 1023.605724] LNet: Removed LNI 192.168.204.2@tcp [ 1024.506167] Key type .llcrypt unregistered [ 1024.508736] Key type ._llcrypt unregistered [ 1037.385145] Key type ._llcrypt registered [ 1037.386766] Key type .llcrypt registered [ 1038.622436] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 12:18:48 (1788970728) [ 1040.481624] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1042.490408] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:18:52 (1788970732) [ 1042.929286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1042.971888] alg: No test for adler32 (adler32-zlib) [ 1043.990565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1044.057242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1044.075737] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1045.004056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.2 [ 1045.030692] LNet: 54446:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1045.038806] LNet: Removed LNI 192.168.204.2@tcp [ 1051.478671] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:19:01 (1788970741) [ 1051.590486] LNet: 55010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.599266] LNetError: 55010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1051.630358] LNet: Removed LNI 192.168.204.2@tcp [ 1052.331385] Key type .llcrypt unregistered [ 1052.333626] Key type ._llcrypt unregistered [ 1058.980877] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1059.245508] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1062.060385] Key type ._llcrypt registered [ 1062.063382] Key type .llcrypt registered [ 1062.261854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1062.288750] alg: No test for adler32 (adler32-zlib) [ 1063.692140] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1071.935639] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:19:21 (1788970761) [ 1072.075378] LNet: 55917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.086411] LNetError: 55917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1072.104067] LNet: Removed LNI 10.1.2.3@tcp [ 1073.323673] Key type .llcrypt unregistered [ 1073.330502] Key type ._llcrypt unregistered [ 1081.476952] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1081.885187] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1081.895330] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1081.905366] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1084.899693] Key type ._llcrypt registered [ 1084.903162] Key type .llcrypt registered [ 1085.234756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1085.259386] alg: No test for adler32 (adler32-zlib) [ 1086.245840] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1093.229055] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:19:42 (1788970782) [ 1094.960496] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1097.160621] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:19:46 (1788970786) [ 1097.285694] LNet: 57103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.290970] LNetError: 57103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1097.316190] LNet: Removed LNI 10.1.2.3@tcp [ 1098.238596] Key type .llcrypt unregistered [ 1098.242726] Key type ._llcrypt unregistered [ 1105.085539] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1105.403940] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1105.418781] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1105.436878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1107.149771] Key type ._llcrypt registered [ 1107.166453] Key type .llcrypt registered [ 1107.418345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1107.434385] alg: No test for adler32 (adler32-zlib) [ 1108.381947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1108.457338] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1117.726992] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:20:07 (1788970807) [ 1118.302560] LNet: 58327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.310444] LNetError: 58327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1118.335743] LNet: Removed LNI 10.1.2.3@tcp [ 1119.179504] Key type .llcrypt unregistered [ 1119.188239] Key type ._llcrypt unregistered [ 1126.241365] Key type ._llcrypt registered [ 1126.243738] Key type .llcrypt registered [ 1126.522278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1126.538564] alg: No test for adler32 (adler32-zlib) [ 1127.531245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1127.588286] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1136.028033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.102@tcp [ 1136.737256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_interrupt -r 1 [ 1136.773694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_interrupt -r 1 [ 1136.831391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1136.849686] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1137.144226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1137.184518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1137.225937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.278827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.311971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1137.337625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.380784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1137.420788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.446785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1137.483312] LNet: 58764:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1137.511711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1137.547377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1137.584337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1137.623738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1137.667247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.708795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1137.767632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1138.472666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_dropped -r 1 [ 1138.549413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_dropped -r 1 [ 1138.614082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1138.649729] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1138.847223] LNet: 1 local NIs in recovery (showing 1): 192.168.204.2@tcp [ 1138.951815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1138.989586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1139.053648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1139.080410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1139.122539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1139.162825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1139.216246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1139.259683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1139.301705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1139.331147] LNet: 58875:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1139.351536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1139.402640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1139.442560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1139.507141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1139.557606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1139.601791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1139.644573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1140.295361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_aborted -r 1 [ 1140.338865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_aborted -r 1 [ 1140.407438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1140.438907] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1140.720418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.747879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1140.775080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.811615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1140.848503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1140.910314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.967210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1141.033838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1141.110926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1141.127768] LNet: 58986:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1141.146214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1141.185210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1141.223571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1141.286419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1141.336244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.390101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1141.483865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1142.093938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_no_route -r 1 [ 1142.143420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_no_route -r 1 [ 1142.182540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1142.470732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1142.507192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1142.570665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1142.610842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1142.646142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1142.696945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1142.732445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1142.782564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1142.821841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1142.853931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1142.887378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1142.921713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1142.962550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1143.011598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1143.061674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1143.128751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1143.746385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_timeout -r 1 [ 1143.808627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_timeout -r 1 [ 1143.861902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1143.893570] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1143.898947] LNet: Skipped 1 previous similar message [ 1144.147865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1144.191978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1144.232367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1144.267777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1144.318374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1144.371077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1144.417106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1144.446483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1144.485301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1144.509446] LNet: 59209:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1144.523818] LNet: 59209:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1144.546456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1144.599885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1144.630438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.657700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.687865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1144.720983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.748606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1145.290778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_error -r 1 [ 1145.318415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_error -r 1 [ 1145.346678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1145.669876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1145.713175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1145.784392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1145.823888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1145.863304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1145.909099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1145.953894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1146.001060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1146.039766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1146.080558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1146.116912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1146.196465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1146.243851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1146.296737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1146.375186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1146.448267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1146.830540] LNet: 59361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1146.842362] LNetError: 59361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1146.855902] LNet: Removed LNI 192.168.204.2@tcp [ 1147.298293] Key type .llcrypt unregistered [ 1147.302146] Key type ._llcrypt unregistered [ 1167.820323] Key type ._llcrypt registered [ 1167.826515] Key type .llcrypt registered [ 1169.193563] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:20:58 (1788970858) [ 1170.348105] Key type .llcrypt unregistered [ 1170.352060] Key type ._llcrypt unregistered [ 1177.074312] Key type ._llcrypt registered [ 1177.084774] Key type .llcrypt registered [ 1177.358068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1177.405714] alg: No test for adler32 (adler32-zlib) [ 1178.360599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1178.406605] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1187.860205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.102@tcp [ 1202.296618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1202.340427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1202.374890] LNet: Added LNI 192.168.204.2@tcp1 [8/256/0/180] [ 1203.944198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_interrupt -r 1 [ 1203.989489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e local_interrupt -r 1 [ 1204.058744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_interrupt -r 1 [ 1204.146575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e local_interrupt -r 1 [ 1204.225724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e local_interrupt -r 1 [ 1204.311219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e local_interrupt -r 1 [ 1204.404375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e local_interrupt -r 1 [ 1204.457086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e local_interrupt -r 1 [ 1204.561808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1204.588275] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1204.601823] LNet: 1 local NIs in recovery (showing 1): 192.168.204.2@tcp [ 1204.847103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1204.869801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1204.893732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1204.918391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1204.950448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1204.976943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1205.005262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1205.032425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1205.060060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1205.075351] LNet: 60829:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1205.090512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1205.120518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1205.177925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1205.226753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1205.274159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.326502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1205.383782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1205.896894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_dropped -r 1 [ 1205.929846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e local_dropped -r 1 [ 1205.990482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_dropped -r 1 [ 1206.046087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e local_dropped -r 1 [ 1206.082476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e local_dropped -r 1 [ 1206.119284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e local_dropped -r 1 [ 1206.171362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e local_dropped -r 1 [ 1206.221571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e local_dropped -r 1 [ 1206.312357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1206.355075] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1206.363658] LNet: Skipped 2 previous similar messages [ 1206.640684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1206.690867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1206.734732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1206.791720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1206.847673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1206.902992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1206.955329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1207.024104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1207.101435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1207.136363] LNet: 60953:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1207.196831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1207.277557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1207.325848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1207.369917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1207.397516] LNet: There was an unexpected network error while writing to 192.168.204.2: rc = -22 [ 1207.412697] LNet: Skipped 3 previous similar messages [ 1207.418679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1207.481565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1207.535586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1208.166500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_aborted -r 1 [ 1208.227745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e local_aborted -r 1 [ 1208.270938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_aborted -r 1 [ 1208.329331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e local_aborted -r 1 [ 1208.386388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e local_aborted -r 1 [ 1208.453063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e local_aborted -r 1 [ 1208.500228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e local_aborted -r 1 [ 1208.533486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e local_aborted -r 1 [ 1208.572441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1208.830467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1208.892496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1208.938841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1209.004975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1209.047917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1209.089208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1209.142424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1209.203469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1209.262940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1209.289351] LNet: 61076:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1209.314752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1209.381868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1209.428603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1209.478380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1209.510766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1209.556500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1209.602221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1210.147281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_no_route -r 1 [ 1210.182668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e local_no_route -r 1 [ 1210.227073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_no_route -r 1 [ 1210.258730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e local_no_route -r 1 [ 1210.297618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e local_no_route -r 1 [ 1210.327335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e local_no_route -r 1 [ 1210.357857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e local_no_route -r 1 [ 1210.405842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e local_no_route -r 1 [ 1210.447765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1210.464872] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1210.469819] LNet: Skipped 3 previous similar messages [ 1210.672864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1210.708047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1210.738613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1210.773262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1210.810880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1210.846979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1210.889417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1210.925830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1210.987240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1211.038178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.078435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1211.123796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.176497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1211.222932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1211.264835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1211.301177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1212.069924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_timeout -r 1 [ 1212.124461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e local_timeout -r 1 [ 1212.196110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_timeout -r 1 [ 1212.263068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e local_timeout -r 1 [ 1212.308736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e local_timeout -r 1 [ 1212.358311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e local_timeout -r 1 [ 1212.420435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e local_timeout -r 1 [ 1212.483667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e local_timeout -r 1 [ 1212.520229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1212.842185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.891620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1212.943498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.989323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1213.066272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1213.121770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1213.167445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1213.240321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1213.306224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1213.348207] LNet: 61323:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1213.355317] LNet: 61323:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1213.382751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1213.425242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1213.473394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1213.503059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1213.579276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.657428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1213.770118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1214.490644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_error -r 1 [ 1214.565509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e local_error -r 1 [ 1214.622863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_error -r 1 [ 1214.687673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e local_error -r 1 [ 1214.740244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e local_error -r 1 [ 1214.800755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e local_error -r 1 [ 1214.848944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e local_error -r 1 [ 1214.902332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e local_error -r 1 [ 1214.932737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1214.972693] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1214.983914] LNet: Skipped 5 previous similar messages [ 1215.277152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1215.309374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1215.371496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1215.422510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1215.500364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1215.552763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1215.589153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1215.643681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1215.692248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1215.745730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1215.814416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1215.854229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1215.921349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1215.966097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1216.006577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1216.043553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1221.882812] LNet: 61523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.887896] LNetError: 61523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.894842] LNet: Removed LNI 192.168.204.2@tcp [ 1222.950881] LNet: Removed LNI 192.168.204.2@tcp1 [ 1223.365940] Key type .llcrypt unregistered [ 1223.369682] Key type ._llcrypt unregistered [ 1241.262110] Key type ._llcrypt registered [ 1241.267226] Key type .llcrypt registered [ 1242.577778] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:22:12 (1788970932) [ 1243.597421] Key type .llcrypt unregistered [ 1243.612259] Key type ._llcrypt unregistered [ 1250.547218] Key type ._llcrypt registered [ 1250.549854] Key type .llcrypt registered [ 1250.766811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1250.783775] alg: No test for adler32 (adler32-zlib) [ 1251.723606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1251.746092] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1259.886869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.102@tcp [ 1260.586868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e remote_dropped -r 1 [ 1260.618613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e remote_dropped -r 1 [ 1260.660381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1260.679852] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1260.998413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1261.035567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1261.069276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1261.099528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1261.163677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1261.223331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1261.289590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1261.324158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1261.356782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1261.378899] LNet: 62770:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1261.417339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1261.468838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1261.505634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1261.569521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1261.625553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1261.691912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1261.746125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1262.425707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e remote_error -r 1 [ 1262.478646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e remote_error -r 1 [ 1262.550426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1262.580926] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1262.920878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1262.961426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1262.987625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1263.015536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1263.069691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1263.131940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1263.174183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1263.241533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1263.280362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1263.301210] LNet: 62881:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1263.317993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1263.350671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1263.384359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1263.432574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1263.479862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1263.517649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1263.569431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1264.278638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e remote_timeout -r 1 [ 1264.331904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e remote_timeout -r 1 [ 1264.369436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1264.396282] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1264.614475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1264.652958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1264.693556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1264.724491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1264.757492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1264.805377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1264.848279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1264.914458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1264.972668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1264.990215] LNet: 62992:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1265.013317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1265.060790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1265.106391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1265.148513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1265.187383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1265.235458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1265.288186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1265.689640] LNet: 63033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1265.696568] LNetError: 63033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1265.715208] LNet: Removed LNI 192.168.204.2@tcp [ 1266.280530] Key type .llcrypt unregistered [ 1266.287078] Key type ._llcrypt unregistered [ 1284.274448] Key type ._llcrypt registered [ 1284.278338] Key type .llcrypt registered [ 1285.746360] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:22:55 (1788970975) [ 1286.927318] Key type .llcrypt unregistered [ 1286.929086] Key type ._llcrypt unregistered [ 1294.057347] Key type ._llcrypt registered [ 1294.063508] Key type .llcrypt registered [ 1294.449788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1294.469452] alg: No test for adler32 (adler32-zlib) [ 1295.422585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1295.450301] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1304.159775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.102@tcp [ 1314.402657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1314.455739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1314.484320] LNet: Added LNI 192.168.204.2@tcp1 [8/256/0/180] [ 1315.862802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e remote_dropped -r 1 [ 1315.916311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e remote_dropped -r 1 [ 1315.974042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e remote_dropped -r 1 [ 1316.026774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e remote_dropped -r 1 [ 1316.071692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e remote_dropped -r 1 [ 1316.135313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e remote_dropped -r 1 [ 1316.175530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e remote_dropped -r 1 [ 1316.229454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e remote_dropped -r 1 [ 1316.273556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1316.294518] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1316.303827] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.102@tcp [ 1316.517905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1316.568445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1316.606489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1316.645550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1316.703229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1316.744799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1316.796485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1316.851831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1316.922616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1316.958680] LNet: 64498:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1316.980966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1317.030419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1317.054742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1317.075792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1317.097990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1317.134660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1317.158872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1317.185451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1317.690498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e remote_error -r 1 [ 1317.750801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e remote_error -r 1 [ 1317.809069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e remote_error -r 1 [ 1317.859594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e remote_error -r 1 [ 1317.935545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e remote_error -r 1 [ 1317.976966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e remote_error -r 1 [ 1318.012774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e remote_error -r 1 [ 1318.066166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e remote_error -r 1 [ 1318.097637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1318.114754] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1318.120523] LNet: Skipped 2 previous similar messages [ 1318.352928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1318.381125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1318.416489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1318.451704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1318.479746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1318.516210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1318.541291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1318.582975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1318.617095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1318.644520] LNet: 64623:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1318.664647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1318.718562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1318.767129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1318.823717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1318.894560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1318.932937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1318.985669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1319.048493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1319.436504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e remote_timeout -r 1 [ 1319.469446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e remote_timeout -r 1 [ 1319.493782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e remote_timeout -r 1 [ 1319.534125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e remote_timeout -r 1 [ 1319.576527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e remote_timeout -r 1 [ 1319.628480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e remote_timeout -r 1 [ 1319.673978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e remote_timeout -r 1 [ 1319.727245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e remote_timeout -r 1 [ 1319.781698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.102@tcp [ 1319.810874] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1320.114383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1320.154480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1320.192234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1320.243955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1320.289556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1320.315389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1320.338638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1320.377459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1320.461706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1320.506954] LNet: 64749:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1320.525879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1320.574537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1320.618703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1320.666220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1320.702926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1320.754226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1320.797524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1320.831638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1325.938841] LNet: 64826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.949631] LNetError: 64826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1325.968457] LNet: Removed LNI 192.168.204.2@tcp [ 1325.988666] LNet: Removed LNI 192.168.204.2@tcp1 [ 1326.462502] Key type .llcrypt unregistered [ 1326.465248] Key type ._llcrypt unregistered [ 1345.049768] Key type ._llcrypt registered [ 1345.053579] Key type .llcrypt registered [ 1346.555353] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:23:56 (1788971036) [ 1348.279268] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1350.307514] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:24:00 (1788971040) [ 1351.226529] Key type .llcrypt unregistered [ 1351.229882] Key type ._llcrypt unregistered [ 1358.821023] Key type ._llcrypt registered [ 1358.822892] Key type .llcrypt registered [ 1359.050594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1359.072379] alg: No test for adler32 (adler32-zlib) [ 1360.104215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1360.155106] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1369.091767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.102@tcp [ 1369.848622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e network_timeout -r 1 [ 1369.909597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e network_timeout -r 1 [ 1370.000156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.102@tcp [ 1370.061601] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1370.402359] LNet: 1 local NIs in recovery (showing 1): 192.168.204.2@tcp [ 1370.474819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1370.522055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1370.579137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1370.657651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1370.741428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1370.795465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1370.846532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1370.910382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1370.968721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1370.997141] LNet: 66368:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1371.050932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1371.115102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1371.180512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1371.251802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1371.323783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1371.392191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1371.430706] LNet: There was an unexpected network error while writing to 192.168.204.2: rc = -22 [ 1371.752802] LNet: 66407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1371.762534] LNetError: 66407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.787166] LNet: Removed LNI 192.168.204.2@tcp [ 1372.219241] Key type .llcrypt unregistered [ 1372.227305] Key type ._llcrypt unregistered [ 1396.292496] Key type ._llcrypt registered [ 1396.300304] Key type .llcrypt registered [ 1396.612657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1396.640255] alg: No test for adler32 (adler32-zlib) [ 1397.651931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1397.701400] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1405.160902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.102@tcp [ 1415.825626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1415.888810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1415.935172] LNet: Added LNI 192.168.204.2@tcp1 [8/256/0/180] [ 1417.640224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e network_timeout -r 1 [ 1417.698476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e network_timeout -r 1 [ 1417.746535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e network_timeout -r 1 [ 1417.786523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e network_timeout -r 1 [ 1417.857961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e network_timeout -r 1 [ 1417.925519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e network_timeout -r 1 [ 1417.977134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e network_timeout -r 1 [ 1418.013254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e network_timeout -r 1 [ 1418.060507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.102@tcp [ 1418.083928] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1418.210455] LNet: 1 local NIs in recovery (showing 1): 192.168.204.2@tcp [ 1418.215726] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.102@tcp [ 1418.338839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1418.396536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1418.453907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1418.506761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1418.583728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1418.622502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1418.671788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1418.707737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1418.752521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1418.771228] LNet: 67285:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1418.796915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1418.863042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1418.942854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1419.026402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1419.097727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1419.162209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1423.997205] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1424.365715] LNet: 67359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1424.370776] LNetError: 67359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1424.379867] LNet: Removed LNI 192.168.204.2@tcp [ 1424.393786] LNet: Removed LNI 192.168.204.2@tcp1 [ 1424.833826] Key type .llcrypt unregistered [ 1424.842326] Key type ._llcrypt unregistered [ 1439.806455] Key type ._llcrypt registered [ 1439.808443] Key type .llcrypt registered [ 1440.991371] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:25:30 (1788971130) [ 1441.249638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1441.259893] alg: No test for adler32 (adler32-zlib) [ 1442.170679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1442.212231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1442.249304] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1442.944411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1442.975988] LNet: Added LNI 192.168.204.2@tcp1 [8/256/0/180] [ 1443.894482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 1443.963734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1444.012452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1444.061753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1444.105391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1444.145393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1444.221596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.2@tcp [ 1444.257745] LNetError: 68456:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.2@tcp to recovery [ 1444.284303] LNet: 1 local NIs in recovery (showing 1): 192.168.204.2@tcp [ 1444.294287] LNet: There was an unexpected network error while writing to 192.168.204.2: rc = -22 [ 1446.367586] LNet: There was an unexpected network error while writing to 192.168.204.2: rc = -22 [ 1446.770126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1450.463447] LNet: There was an unexpected network error while writing to 192.168.204.2: rc = -22 [ 1451.628984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1451.673251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1451.738535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1451.771770] LNet: 68563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.792862] LNetError: 68563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.806306] LNet: Removed LNI 192.168.204.2@tcp [ 1451.813320] LNet: Removed LNI 192.168.204.2@tcp1 [ 1451.834951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.896858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1451.931710] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1452.827513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1452.851906] LNet: Added LNI 192.168.204.2@tcp1 [8/256/0/180] [ 1453.801584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 1453.847887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1453.928558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1453.992385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1454.036226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1454.096175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1454.148792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1454.185524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.2@tcp [ 1454.211474] LNetError: 68835:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.2@tcp to recovery [ 1454.944277] LNet: There was an unexpected network error while writing to 192.168.204.2: rc = -22 [ 1457.719832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1460.064507] LNet: There was an unexpected network error while writing to 192.168.204.2: rc = -22 [ 1460.078051] LNet: Skipped 1 previous similar message [ 1464.682230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1464.719502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1464.765526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1471.636362] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:26:01 (1788971161) [ 1471.748116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1471.784815] LNet: 69538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.795612] LNetError: 69538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1471.807615] LNetError: 69538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1471.819791] LNet: Removed LNI 192.168.204.2@tcp [ 1471.863302] LNet: Removed LNI 192.168.204.2@tcp1 [ 1471.897802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.970625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1472.009522] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1472.735813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1472.756421] LNet: Added LNI 192.168.204.2@tcp1 [8/256/0/180] [ 1473.709609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 1473.792297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1473.820526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1473.864484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1473.918725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1473.970855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1474.019617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.2@tcp --health 0 [ 1474.116477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1475.039395] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.2@tcp [ 1475.041170] LNet: There was an unexpected network error while writing to 192.168.204.2: rc = -22 [ 1475.061227] LNet: Skipped 1 previous similar message [ 1489.672555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1489.848865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1489.899150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1489.958342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.2@tcp --health 500 [ 1490.074916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1495.698776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1495.759556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1495.779505] LNet: 70028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1495.786411] LNetError: 70028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1495.791371] LNetError: 70028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1495.796361] LNet: Removed LNI 192.168.204.2@tcp [ 1495.805707] LNet: Removed LNI 192.168.204.2@tcp1 [ 1495.828494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1495.884157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1495.916738] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1496.646416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1496.679319] LNet: Added LNI 192.168.204.2@tcp1 [8/256/0/180] [ 1497.539065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 1497.601882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1497.672111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1497.710899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1497.754686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1497.811776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1497.850090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1497.898862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.2@tcp --health 0 [ 1498.913224] LNet: There was an unexpected network error while writing to 192.168.204.2: rc = -22 [ 1498.915945] LNet: Skipped 3 previous similar messages [ 1499.240830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1508.442564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1508.471311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1508.496302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1514.754800] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:26:44 (1788971204) [ 1515.237957] LNet: 71064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1515.260212] LNetError: 71064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1515.269830] LNetError: 71064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1515.275635] LNet: Removed LNI 192.168.204.2@tcp [ 1515.283624] LNet: Removed LNI 192.168.204.2@tcp1 [ 1516.012864] Key type .llcrypt unregistered [ 1516.019325] Key type ._llcrypt unregistered [ 1521.993589] Key type ._llcrypt registered [ 1521.995435] Key type .llcrypt registered [ 1522.222742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1522.240191] alg: No test for adler32 (adler32-zlib) [ 1523.164441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1523.210813] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1532.824409] Lustre: DEBUG MARKER: Initial discovery [ 1532.887907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.102@tcp [ 1535.206443] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1535.390954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1535.420517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.102@tcp [ 1535.437251] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1535.454806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1535.485491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.102@tcp -r 1 -m GET -l 3 [ 1535.514926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.102@tcp -r 1 -m GET -e local_error [ 1535.519102] LNet: 1 local NIs in recovery (showing 1): 192.168.204.2@tcp [ 1535.553414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.102@tcp -r 1 -m PUT -l 6 [ 1536.951383] Lustre: DEBUG MARKER: Force oleg402-client.virtnet to discover 192.168.204.102@tcp (in background) [ 1537.132561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.102@tcp [ 1538.407942] Lustre: DEBUG MARKER: Force oleg402-server to discover 192.168.204.2@tcp [ 1540.191732] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1541.210878] Lustre: DEBUG MARKER: Wait for 71824 [ 1542.519797] Lustre: DEBUG MARKER: Finished wait on 71824 [ 1542.796147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1542.829522] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1542.852269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1543.346167] LNet: 72273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.356321] LNetError: 72273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.423858] LNet: Removed LNI 192.168.204.2@tcp [ 1544.814791] Key type .llcrypt unregistered [ 1544.817285] Key type ._llcrypt unregistered [ 1559.877724] Key type ._llcrypt registered [ 1559.884648] Key type .llcrypt registered [ 1562.025972] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:27:30 (1788971250) [ 1562.719753] Key type .llcrypt unregistered [ 1562.723909] Key type ._llcrypt unregistered [ 1569.987711] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1570.284214] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1570.288701] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1570.305443] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1573.060034] Key type ._llcrypt registered [ 1573.061890] Key type .llcrypt registered [ 1573.212563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1573.230300] alg: No test for adler32 (adler32-zlib) [ 1574.175046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.249271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1574.281866] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1575.287205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1575.325510] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1584.682156] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:27:54 (1788971274) [ 1584.875447] LNet: 74364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1584.892870] LNetError: 74364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1584.901385] LNet: Removed LNI 10.1.2.3@tcp [ 1584.917343] LNet: Removed LNI 192.168.204.2@tcp [ 1585.535462] Key type .llcrypt unregistered [ 1585.540827] Key type ._llcrypt unregistered [ 1593.434468] Key type ._llcrypt registered [ 1593.436822] Key type .llcrypt registered [ 1593.727441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1593.746817] alg: No test for adler32 (adler32-zlib) [ 1594.677302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.751450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1594.783896] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1595.534346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1595.554104] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1596.802502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.2@tcp [ 1597.855255] LNetError: Unexpected error -22 connecting to 192.168.204.2@tcp at host 192.168.204.2:988 [ 1603.314937] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:28:13 (1788971293) [ 1603.429928] LNet: 75485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.442038] LNetError: 75485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1603.453175] LNet: Removed LNI 10.1.2.3@tcp [ 1603.477220] LNet: Removed LNI 192.168.204.2@tcp [ 1604.147422] Key type .llcrypt unregistered [ 1604.149567] Key type ._llcrypt unregistered [ 1609.230921] Key type ._llcrypt registered [ 1609.232215] Key type .llcrypt registered [ 1609.336188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1609.344572] alg: No test for adler32 (adler32-zlib) [ 1610.257255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.334702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1610.368750] LNet: Added LNI 192.168.204.2@tcp1 [8/256/0/180] [ 1611.117427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1611.130425] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 1611.759981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.2@tcp1 --nid 192.168.204.2@tcp2 [ 1615.605106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.2@tcp1 192.168.204.2@tcp2 [ 1615.637173] LNetError: 76327:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.2@tcp1 to 192.168.204.2@tcp2 - Route table may be misconfigured [ 1615.667456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.2@tcp2 192.168.204.2@tcp1 [ 1621.711972] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:28:31 (1788971311) [ 1621.835120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.863851] LNet: 76894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.871751] LNetError: 76894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1621.886871] LNet: Removed LNI 192.168.204.2@tcp1 [ 1621.912419] LNet: Removed LNI 192.168.204.2@tcp2 [ 1621.982830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.087939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1622.152601] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1623.272781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1623.306361] LNet: Added LNI 192.168.204.2@tcp1 [8/256/0/180] [ 1624.483399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 1624.541751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.2@tcp 192.168.204.2@tcp [ 1624.604821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.2@tcp1 192.168.204.2@tcp1 [ 1624.715329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.2@tcp -d 192.168.204.2@tcp -e network_timeout [ 1624.796288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -e network_timeout [ 1624.861302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -e network_timeout [ 1624.945100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -e network_timeout [ 1625.014668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 1625.051192] LNet: There was an unexpected network error while writing to 192.168.204.2: rc = -22 [ 1625.095601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1625.120770] LNet: 1 local NIs in recovery (showing 1): 192.168.204.2@tcp [ 1625.215898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1625.437062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1634.766631] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:28:44 (1788971324) [ 1634.900301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.975322] LNet: 77758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.984797] LNetError: 77758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.995169] LNetError: 77758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1635.003546] LNet: Removed LNI 192.168.204.2@tcp [ 1636.075180] LNet: Removed LNI 192.168.204.2@tcp1 [ 1636.215893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.556632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1636.985263] LNet: 77800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1637.906828] Key type .llcrypt unregistered [ 1637.911735] Key type ._llcrypt unregistered [ 1651.622864] Key type ._llcrypt registered [ 1651.629521] Key type .llcrypt registered [ 1653.104752] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:29:02 (1788971342) [ 1653.895770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1653.922590] alg: No test for adler32 (adler32-zlib) [ 1654.876924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.997221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1657.651672] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:29:07 (1788971347) [ 1657.924155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1657.946430] LNet: 78896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.970599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.021231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1658.041647] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1658.804021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1658.846823] LNet: Added LNI 192.168.204.2@tcp1 [8/256/0/180] [ 1659.737997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 1659.781535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp1 [ 1659.806865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp1 [ 1665.651326] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:29:15 (1788971355) [ 1667.370646] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg402-server [ 1669.515970] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:29:19 (1788971359) [ 1671.480296] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg402-server [ 1673.353984] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:29:23 (1788971363) [ 1675.311634] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg402-server [ 1677.661333] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:29:27 (1788971367) [ 1679.785654] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg402-server [ 1681.657927] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:29:31 (1788971371) [ 1683.524674] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg402-server [ 1685.111662] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:29:35 (1788971375) [ 1686.692641] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg402-server [ 1688.223442] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:29:38 (1788971378) [ 1690.260498] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg402-server [ 1691.837847] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:29:41 (1788971381) [ 1693.463233] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg402-server [ 1695.231355] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:29:45 (1788971385) [ 1697.131411] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg402-server [ 1698.757115] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:29:48 (1788971388) [ 1698.864132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.883068] LNet: 82770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1698.899509] LNetError: 82770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1698.916527] LNet: Removed LNI 192.168.204.2@tcp [ 1699.944639] LNet: Removed LNI 192.168.204.2@tcp1 [ 1700.002736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.088313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1700.122901] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1700.875637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1700.944299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 1701.079580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1701.129177] LNet: 82921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.140662] LNetError: 82921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1701.153049] LNetError: 82921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1701.172550] LNet: Removed LNI 192.168.204.2@tcp [ 1701.203975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.262696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1701.295210] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1701.978831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 1702.166816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.183759] LNetError: 83066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.202700] LNet: Removed LNI 192.168.204.2@tcp [ 1702.231291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.288830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1702.312533] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1703.056678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1703.136548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 1703.285080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.318337] LNet: 83213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.322312] LNet: 83213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1704.368094] LNet: Removed LNI 192.168.204.2@tcp [ 1704.408285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.450494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1704.478935] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1705.189714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 1705.432488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.477457] LNetError: 83358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.481779] LNetError: 83358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1705.497049] LNet: Removed LNI 192.168.204.2@tcp [ 1705.538282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.593231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1705.615234] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1706.323908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1706.413765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 1706.607657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.685157] LNet: Removed LNI 192.168.204.2@tcp [ 1707.747675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.835887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1707.891866] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1708.926909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 1710.385348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.417083] LNet: 83660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.424328] LNet: 83660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1710.453242] LNetError: 83660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.463531] LNetError: 83660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1711.527687] LNet: Removed LNI 192.168.204.2@tcp [ 1711.568855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.665088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1711.693264] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1712.477389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1712.567319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 1712.803465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.896325] LNet: Removed LNI 192.168.204.2@tcp [ 1713.920560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.992545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1714.033762] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1714.881808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 1716.339548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1717.414044] LNet: Removed LNI 192.168.204.2@tcp [ 1717.445929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.501997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1717.540712] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1718.297839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1718.388677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 1718.674799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1718.712317] LNet: 84110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.726739] LNet: 84110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1718.747357] LNetError: 84110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1718.756333] LNetError: 84110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1719.788018] LNet: Removed LNI 192.168.204.2@tcp [ 1719.832389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.909684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1719.946510] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1720.602875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 1720.815770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.893663] LNet: Removed LNI 192.168.204.2@tcp [ 1721.944589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.018377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1722.051256] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1722.785651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1722.862861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 1723.113176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.203085] LNet: Removed LNI 192.168.204.2@tcp [ 1724.240929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.321315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1724.354051] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1725.280712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 1725.618714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.695459] LNet: Removed LNI 192.168.204.2@tcp [ 1726.729709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.828699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1726.860802] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1727.719163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1727.787435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.813644] LNet: Removed LNI 192.168.204.2@tcp [ 1727.830610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.888699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1727.909250] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1728.687924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1728.772198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.805244] LNet: Removed LNI 192.168.204.2@tcp [ 1728.851513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.921353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1729.040586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.092806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.208420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1729.366508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.420531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.480379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1729.575704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.637961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.720464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1729.799099] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1730.741867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.798550] LNet: Removed LNI 192.168.204.2@tcp [ 1730.823765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.904300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1737.780963] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:30:27 (1788971427) [ 1737.858045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.887427] LNet: 85562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1737.894721] LNet: 85562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1737.927770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.024397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1738.060314] LNet: Added LNI 192.168.204.2@tcp231 [8/256/0/180] [ 1739.379518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.393190] LNetError: 85708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.396600] LNetError: 85708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1739.406627] LNet: Removed LNI 192.168.204.2@tcp231 [ 1739.422202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.537802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1739.575685] LNet: Added LNI 192.168.204.2@tcp231 [8/256/0/180] [ 1739.583636] LNet: 85716:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1740.083702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1740.109378] LNet: Removed LNI 192.168.204.2@tcp231 [ 1740.138965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1740.169134] LNet: Added LNI 192.168.204.2@tcp231 [8/256/0/180] [ 1741.499798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.550976] LNet: Removed LNI 192.168.204.2@tcp231 [ 1741.581790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.684679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1741.715573] LNet: Added LNI 192.168.204.2@tcp231 [8/256/0/0] [ 1741.722695] LNet: 85868:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1742.255184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1742.291554] LNet: Removed LNI 192.168.204.2@tcp231 [ 1742.306751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1742.333964] LNet: Added LNI 192.168.204.2@tcp231 [8/256/0/0] [ 1743.959393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.010103] LNet: Removed LNI 192.168.204.2@tcp231 [ 1744.043235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.157719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1744.211537] LNet: Added LNI 192.168.204.2@tcp231 [8/256/0/60] [ 1744.222376] LNet: 86020:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1744.609499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1744.660478] LNet: Removed LNI 192.168.204.2@tcp231 [ 1744.684871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1744.725842] LNet: Added LNI 192.168.204.2@tcp231 [8/256/0/60] [ 1745.810130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.870214] LNet: Removed LNI 192.168.204.2@tcp231 [ 1745.898055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.002588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1746.035116] LNet: Added LNI 192.168.204.2@tcp231 [8/256/0/180] [ 1746.448431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1746.488706] LNet: Removed LNI 192.168.204.2@tcp231 [ 1746.526924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1746.603104] LNet: Added LNI 192.168.204.2@tcp231 [8/256/0/180] [ 1755.391192] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:30:45 (1788971445) [ 1755.535525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.627937] LNet: Removed LNI 192.168.204.2@tcp231 [ 1755.672474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.764967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1755.796599] LNet: Added LNI 192.168.204.2@tcp232 [8/256/0/180] [ 1756.671688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1756.733775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1756.762804] LNet: Removed LNI 192.168.204.2@tcp232 [ 1756.777371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1756.804951] LNet: Added LNI 192.168.204.2@tcp232 [8/256/0/180] [ 1757.627684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1757.683750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1757.719933] LNet: Removed LNI 192.168.204.2@tcp232 [ 1757.735801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1757.761292] LNet: Added LNI 192.168.204.2@tcp232 [8/256/0/180] [ 1758.557963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1758.593832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1758.625929] LNet: Removed LNI 192.168.204.2@tcp232 [ 1758.648233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1758.673744] LNet: Added LNI 192.168.204.2@tcp232 [8/256/0/180] [ 1759.357797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1759.390832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1759.419931] LNet: Removed LNI 192.168.204.2@tcp232 [ 1759.432745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1765.404653] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:30:55 (1788971455) [ 1766.481869] Key type .llcrypt unregistered [ 1766.495144] Key type ._llcrypt unregistered [ 1772.506739] Key type ._llcrypt registered [ 1772.508878] Key type .llcrypt registered [ 1772.685628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1772.705846] alg: No test for adler32 (adler32-zlib) [ 1773.654879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1773.685989] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1779.810657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.102@tcp [ 1787.540662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1787.591936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1787.612888] LNet: Added LNI 192.168.204.2@tcp1 [8/256/0/180] [ 1789.006844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_interrupt -r 2 [ 1789.049436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e local_interrupt -r 2 [ 1789.081447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_interrupt -r 2 [ 1789.113751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e local_interrupt -r 2 [ 1789.156932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e local_interrupt -r 2 [ 1789.206129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e local_interrupt -r 2 [ 1789.247886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e local_interrupt -r 2 [ 1789.296517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e local_interrupt -r 2 [ 1789.352658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.2@tcp 192.168.204.102@tcp [ 1789.370173] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1789.374496] LNet: 1 local NIs in recovery (showing 1): 192.168.204.2@tcp [ 1789.613169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1789.651911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1789.687333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1789.726182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1789.772092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1789.804800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1789.832308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1789.876746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1789.915461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1789.944552] LNet: 88673:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1789.974230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1790.005384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1790.042779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1790.080360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1790.124987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1790.179951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1790.691085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_dropped -r 2 [ 1790.730270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e local_dropped -r 2 [ 1790.767452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_dropped -r 2 [ 1790.824986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e local_dropped -r 2 [ 1790.877961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e local_dropped -r 2 [ 1790.935098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e local_dropped -r 2 [ 1790.984431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e local_dropped -r 2 [ 1791.026350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e local_dropped -r 2 [ 1791.058808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.2@tcp 192.168.204.102@tcp [ 1791.078739] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1791.305938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1791.331274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1791.372286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1791.404198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1791.435609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1791.473494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1791.509312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1791.538887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1791.576744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1791.599964] LNet: 88795:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1791.617812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1791.649040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1791.675848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1791.710910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1791.770991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1791.816532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1792.350396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_aborted -r 2 [ 1792.383246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e local_aborted -r 2 [ 1792.428619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_aborted -r 2 [ 1792.472473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e local_aborted -r 2 [ 1792.503408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e local_aborted -r 2 [ 1792.563628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e local_aborted -r 2 [ 1792.610856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e local_aborted -r 2 [ 1792.653255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e local_aborted -r 2 [ 1792.707465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.2@tcp 192.168.204.102@tcp [ 1792.727441] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1792.997233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1793.019466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1793.044131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1793.069453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1793.099991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1793.140996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1793.198729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1793.246215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1793.302625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1793.323509] LNet: 88917:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1793.343740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1793.370980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1793.402654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1793.442186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1793.483270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1793.512413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1794.039990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_no_route -r 2 [ 1794.078146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e local_no_route -r 2 [ 1794.125788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_no_route -r 2 [ 1794.155775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e local_no_route -r 2 [ 1794.197974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e local_no_route -r 2 [ 1794.232487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e local_no_route -r 2 [ 1794.283609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e local_no_route -r 2 [ 1794.336677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e local_no_route -r 2 [ 1794.431742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.2@tcp 192.168.204.102@tcp [ 1794.705762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1794.736765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1794.761515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1794.793416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1794.820363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1794.844385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1794.879663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1794.935436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1794.962282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1794.993307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1795.025423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1795.056925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1795.083696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1795.111869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1795.144571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1795.615935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e local_timeout -r 2 [ 1795.659748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e local_timeout -r 2 [ 1795.702566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e local_timeout -r 2 [ 1795.732032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e local_timeout -r 2 [ 1795.764287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e local_timeout -r 2 [ 1795.800624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e local_timeout -r 2 [ 1795.833903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e local_timeout -r 2 [ 1795.886556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e local_timeout -r 2 [ 1795.948172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.2@tcp 192.168.204.102@tcp [ 1795.981659] LNet: There was an unexpected network error while writing to 192.168.204.102: rc = -22 [ 1795.989351] LNet: Skipped 1 previous similar message [ 1796.260901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1796.300629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1796.338898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1796.378264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1796.425770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1796.504712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1796.560001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1796.590955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1796.639552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1796.662659] LNet: 89162:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1796.669705] LNet: 89162:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1796.710889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1796.776397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1796.806106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1796.831671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1796.857719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1796.883102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1797.388374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp -m GET -e remote_dropped -r 2 [ 1797.427887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.102@tcp1 -m GET -e remote_dropped -r 2 [ 1797.462332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp -m GET -e remote_dropped -r 2 [ 1797.499983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp -d 192.168.204.2@tcp1 -m GET -e remote_dropped -r 2 [ 1797.547901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp -m GET -e remote_dropped -r 2 [ 1797.612607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.2@tcp1 -d 192.168.204.102@tcp1 -m GET -e remote_dropped -r 2 [ 1797.663292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp -m GET -e remote_dropped -r 2 [ 1797.727263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.2@tcp1 -d 192.168.204.2@tcp1 -m GET -e remote_dropped -r 2 [ 1797.767952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.2@tcp 192.168.204.102@tcp [ 1797.810969] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.102@tcp [ 1798.092119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1798.119587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1798.149125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1798.172430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1798.198431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1798.237567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1798.265167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1798.299540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1798.319922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1798.346216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1798.369263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1798.397652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1798.424639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1798.452165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1798.481920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1804.503605] LNet: 89359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.515123] LNetError: 89359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.524891] LNet: Removed LNI 192.168.204.2@tcp [ 1804.544462] LNet: Removed LNI 192.168.204.2@tcp1 [ 1804.917662] Key type .llcrypt unregistered [ 1804.921270] Key type ._llcrypt unregistered [ 1822.244179] Key type ._llcrypt registered [ 1822.248067] Key type .llcrypt registered [ 1823.850374] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:31:53 (1788971513) [ 1824.316826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.329149] alg: No test for adler32 (adler32-zlib) [ 1825.247457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.314217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1825.364408] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1826.250359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1826.301131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.2@tcp [ 1826.331336] LNetError: 90338:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.2@tcp to recovery [ 1826.430335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.2@tcp [ 1826.539302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1833.986451] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:32:03 (1788971523) [ 1834.129632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.152818] LNet: 90925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.163563] LNetError: 90925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1834.188654] LNet: Removed LNI 192.168.204.2@tcp [ 1834.225501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.280502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1834.305066] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1835.137581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1835.163262] LNet: Added LNI 192.168.204.2@tcp1 [8/256/0/180] [ 1836.088950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.2@tcp [ 1836.120872] LNetError: 91194:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.2@tcp to recovery [ 1836.321728] LNet: 1 local NIs in recovery (showing 1): 192.168.204.2@tcp [ 1836.329677] LNet: There was an unexpected network error while writing to 192.168.204.2: rc = -22 [ 1837.349487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.2@tcp [ 1844.856790] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:32:14 (1788971534) [ 1846.487555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg402-server [ 1848.536439] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:32:18 (1788971538) [ 1850.174493] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg402-server [ 1851.581752] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:32:21 (1788971541) [ 1853.675461] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg402-server [ 1855.502496] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:32:25 (1788971545) [ 1855.816260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1855.830276] LNet: 92815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.854924] LNetError: 92815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.866080] LNet: Removed LNI 192.168.204.2@tcp [ 1855.891151] LNet: Removed LNI 192.168.204.2@tcp1 [ 1855.940348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.022047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1856.064310] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1857.064857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1857.125341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1857.177361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1857.305409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1857.363322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1857.473536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1863.697782] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:32:33 (1788971553) [ 1863.936756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1863.983933] LNet: 93544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1863.997177] LNetError: 93544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.017167] LNetError: 93544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1864.050800] LNet: Removed LNI 192.168.204.2@tcp [ 1864.102448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1864.174844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1864.209097] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1872.245980] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:32:41 (1788971561) [ 1872.346365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.375880] LNet: 94245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.385367] LNetError: 94245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.403498] LNet: Removed LNI 192.168.204.2@tcp [ 1872.443566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.495989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1872.517707] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1879.621704] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:32:49 (1788971569) [ 1880.941769] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1882.357948] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:32:52 (1788971572) [ 1882.961632] LNet: 95286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.966493] LNetError: 95286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.979468] LNet: Removed LNI 192.168.204.2@tcp [ 1883.481467] Key type .llcrypt unregistered [ 1883.482925] Key type ._llcrypt unregistered [ 1889.802663] Key type ._llcrypt registered [ 1889.805440] Key type .llcrypt registered [ 1889.998915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1890.016639] alg: No test for adler32 (adler32-zlib) [ 1890.973961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1890.998111] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1898.551617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.102@tcp [ 1903.873439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.102@tcp [ 1904.269299] LNet: 95701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.279876] LNetError: 95701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1904.295660] LNet: Removed LNI 192.168.204.2@tcp [ 1904.754606] Key type .llcrypt unregistered [ 1904.761348] Key type ._llcrypt unregistered [ 1916.626876] Key type ._llcrypt registered [ 1916.628559] Key type .llcrypt registered [ 1918.082898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1919.543972] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1920.805603] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:33:31 (1788971611) [ 1922.200723] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1923.739437] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:33:33 (1788971613) [ 1925.444975] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg402-server [ 1927.176332] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:33:37 (1788971617) [ 1928.920992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg402-server [ 1930.840396] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:33:40 (1788971620) [ 1932.060729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1932.103658] alg: No test for adler32 (adler32-zlib) [ 1933.248985] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1933.513097] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1935.183604] Key type lgssc registered [ 1936.731224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1957.805184] Key type lgssc unregistered [ 1958.038664] LNet: 98872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.052194] LNetError: 98872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.080788] LNet: Removed LNI 192.168.204.2@tcp [ 1959.023577] Key type .llcrypt unregistered [ 1959.025047] Key type ._llcrypt unregistered [ 1978.499621] Key type ._llcrypt registered [ 1978.503818] Key type .llcrypt registered [ 1979.955851] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:34:29 (1788971669) [ 1980.626119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1980.658451] alg: No test for adler32 (adler32-zlib) [ 1981.626722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.738058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1981.816777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1981.896720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1981.950224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1989.687176] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:34:39 (1788971679) [ 1989.808314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1989.843444] LNet: 100280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.885527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.979334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1990.026161] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1991.074736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1991.121748] LNet: 100431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.129165] LNetError: 100431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1991.147071] LNet: Removed LNI 192.168.204.2@tcp [ 1991.162135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1991.209114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1991.230634] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1991.630409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1991.652908] LNetError: 100459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1991.671387] LNet: Removed LNI 192.168.204.2@tcp [ 1991.713893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1991.782689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1991.839495] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 1998.858317] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:34:48 (1788971688) [ 1999.084747] LNet: 101050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1999.091403] LNet: 101050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1999.101772] LNetError: 101050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1999.120342] LNet: Removed LNI 192.168.204.2@tcp [ 1999.779729] Key type .llcrypt unregistered [ 1999.783637] Key type ._llcrypt unregistered [ 2006.014793] Key type ._llcrypt registered [ 2006.022705] Key type .llcrypt registered [ 2006.215270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2006.224565] alg: No test for adler32 (adler32-zlib) [ 2007.080352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.149332] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2007.152799] LNet: *** cfs_fail_loc=e003, val=100*** [ 2007.990112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2008.010268] LNet: *** cfs_fail_loc=e003, val=100*** [ 2008.015044] LNet: Skipped 2 previous similar messages [ 2008.034144] Ping reply received with mapping: [ 2008.035325] found metadata reply NI[192.168.204.2@tcp] [ 2008.037467] 02 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2008.040575] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2008.043140] 5a 5a 5a 5a 5a 5a 5a 5a [ 2008.152773] LNet: 101444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2008.170241] LNetError: 101444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2008.181552] LNet: Removed LNI 192.168.204.2@tcp [ 2008.880545] Key type .llcrypt unregistered [ 2008.883622] Key type ._llcrypt unregistered [ 2022.149552] Key type ._llcrypt registered [ 2022.153220] Key type .llcrypt registered [ 2023.683626] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:35:13 (1788971713) [ 2024.205299] Key type .llcrypt unregistered [ 2024.206918] Key type ._llcrypt unregistered [ 2032.334854] Key type ._llcrypt registered [ 2032.338501] Key type .llcrypt registered [ 2032.709542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2032.738411] alg: No test for adler32 (adler32-zlib) [ 2034.033193] LNetError: 102482:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2034.622698] Key type .llcrypt unregistered [ 2034.624282] Key type ._llcrypt unregistered [ 2041.289174] Key type ._llcrypt registered [ 2041.293377] Key type .llcrypt registered [ 2043.343811] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:35:32 (1788971732) [ 2044.309284] Key type .llcrypt unregistered [ 2044.310948] Key type ._llcrypt unregistered [ 2051.930912] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2051.941477] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2054.489225] Key type ._llcrypt registered [ 2054.490680] Key type .llcrypt registered [ 2054.731808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2054.749913] alg: No test for adler32 (adler32-zlib) [ 2055.647388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2055.693306] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2065.348366] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:35:55 (1788971755) [ 2065.553298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2065.599249] LNet: 104256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2065.615189] LNetError: 104256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2065.652221] LNet: Removed LNI 10.1.2.3@tcp [ 2065.709592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2066.529764] Key type .llcrypt unregistered [ 2066.531522] Key type ._llcrypt unregistered [ 2074.256263] Key type ._llcrypt registered [ 2074.259038] Key type .llcrypt registered [ 2074.443415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2074.469859] alg: No test for adler32 (adler32-zlib) [ 2075.426844] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2075.510571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2075.730080] LNet: 104542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2075.743555] LNetError: 104542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2075.769224] LNet: Removed LNI 192.168.204.2@tcp [ 2076.549498] Key type .llcrypt unregistered [ 2076.551955] Key type ._llcrypt unregistered [ 2088.690079] Key type ._llcrypt registered [ 2088.693072] Key type .llcrypt registered [ 2090.242938] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:36:19 (1788971779) [ 2091.012492] Key type .llcrypt unregistered [ 2091.017767] Key type ._llcrypt unregistered [ 2098.062748] Key type ._llcrypt registered [ 2098.065566] Key type .llcrypt registered [ 2098.242928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2098.269489] alg: No test for adler32 (adler32-zlib) [ 2106.027541] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:36:35 (1788971795) [ 2106.266707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2106.335928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.484809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2106.511545] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2107.449117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2107.509160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2107.551857] LNet: Removed LNI 192.168.204.2@tcp [ 2107.811423] LNet: 106284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.333664] Key type .llcrypt unregistered [ 2108.337322] Key type ._llcrypt unregistered [ 2119.673349] Key type ._llcrypt registered [ 2119.679274] Key type .llcrypt registered [ 2120.953521] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:36:50 (1788971810) [ 2121.986307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2122.008571] alg: No test for adler32 (adler32-zlib) [ 2123.117420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.178666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2123.213229] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2124.204468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2124.277799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.2@tcp [ 2124.293694] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2124.296901] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2124.300148] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2124, 0, 0 [ 2124.304881] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2124.308664] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2124.313186] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2124.316454] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2124, 0, 0 [ 2124.320943] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2124.326222] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2124.330267] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2124.334185] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2124, 0, 2174 [ 2124.338848] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2124.342331] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2124.347717] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2124.351885] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2124, 0, 2174 [ 2124.357627] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2132.391665] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:37:02 (1788971822) [ 2133.026166] LNet: 107822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2133.036926] LNetError: 107822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.061053] LNet: Removed LNI 192.168.204.2@tcp [ 2133.830093] Key type .llcrypt unregistered [ 2133.836443] Key type ._llcrypt unregistered [ 2142.000563] Key type ._llcrypt registered [ 2142.002688] Key type .llcrypt registered [ 2142.369254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2142.403825] alg: No test for adler32 (adler32-zlib) [ 2143.329843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2143.367426] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2152.490447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.102@tcp [ 2164.125550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2164.178800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2164.212461] LNet: Added LNI 192.168.204.2@tcp1 [8/256/0/180] [ 2165.387331] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2165.627647] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2165.640775] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2165.644087] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2168.106424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2168.145635] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2178.646670] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:37:48 (1788971868) [ 2178.861195] LNet: 109328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2178.870975] LNetError: 109328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2178.880424] LNet: Removed LNI 192.168.204.2@tcp [ 2179.940272] LNet: Removed LNI 192.168.204.2@tcp1 [ 2179.947841] LNetError: 109328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2179.961692] LNetError: 109328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2179.985663] LNet: Removed LNI 10.1.2.3@tcp99 [ 2180.759696] Key type .llcrypt unregistered [ 2180.762113] Key type ._llcrypt unregistered [ 2187.476539] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2187.480305] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2190.128690] Key type ._llcrypt registered [ 2190.131671] Key type .llcrypt registered [ 2190.419697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2190.438919] alg: No test for adler32 (adler32-zlib) [ 2191.384937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2191.485491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2191.560529] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2192.429123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2192.458563] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2195.046209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2195.105086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2195.168699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2195.213396] LNetError: 110082:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2195.261114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2195.317986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2195.902317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2196.009713] LNet: 110102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.026450] LNetError: 110102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2196.035607] LNet: Removed LNI 10.1.2.3@tcp [ 2197.094934] LNet: Removed LNI 192.168.204.2@tcp [ 2197.812682] Key type .llcrypt unregistered [ 2197.817371] Key type ._llcrypt unregistered [ 2210.750969] Key type ._llcrypt registered [ 2210.752546] Key type .llcrypt registered [ 2212.381794] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:38:21 (1788971901) [ 2212.897984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2212.906669] alg: No test for adler32 (adler32-zlib) [ 2213.868258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2213.914161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2213.929423] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2214.756366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2214.907674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg402-client@tcp [ 2222.900581] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:38:32 (1788971912) [ 2223.016758] LNet: 111630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2223.024979] LNetError: 111630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2223.063605] LNet: Removed LNI 192.168.204.2@tcp [ 2224.059466] Key type .llcrypt unregistered [ 2224.064783] Key type ._llcrypt unregistered [ 2230.976259] Key type ._llcrypt registered [ 2230.988295] Key type .llcrypt registered [ 2231.411921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2231.456617] alg: No test for adler32 (adler32-zlib) [ 2232.407277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2232.439269] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2232.474693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2240.302715] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:38:49 (1788971929) [ 2242.271414] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2244.576885] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:38:54 (1788971934) [ 2246.287972] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2248.272912] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:38:57 (1788971937) [ 2248.493748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2248.546946] LNet: 113052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.561328] LNetError: 113052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.594964] LNet: Removed LNI 192.168.204.2@tcp [ 2248.630511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.706525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2248.745939] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2249.833963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2249.914064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2256.362798] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:39:06 (1788971946) [ 2256.560245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.585307] LNet: 113765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2256.610688] LNetError: 113765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2256.626676] LNet: Removed LNI 192.168.204.2@tcp [ 2256.640632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.688518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2256.731423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2257.473527] Key type .llcrypt unregistered [ 2257.481100] Key type ._llcrypt unregistered [ 2268.833174] Key type ._llcrypt registered [ 2268.842246] Key type .llcrypt registered [ 2270.391417] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:39:19 (1788971959) [ 2270.596374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2270.621111] alg: No test for adler32 (adler32-zlib) [ 2271.638866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.715125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2271.765771] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2273.005495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2273.098924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2284.901072] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:39:33 (1788971973) [ 2285.222758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2285.298147] LNet: 115318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2285.320207] LNetError: 115318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2285.343050] LNet: Removed LNI 192.168.204.2@tcp [ 2285.436562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2285.553755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2285.617298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2293.469520] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:39:43 (1788971983) [ 2293.652105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2293.702318] LNet: 115896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.757827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.885493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2294.450405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2294.515830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2294.645843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2295.214684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2295.287918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2295.390798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2295.424927] LNet: 115933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2295.447125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.553705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2296.045211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2296.071851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2296.115189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2296.406611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2296.429533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2303.032131] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:39:53 (1788971993) [ 2303.215849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2303.230171] LNet: 116535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2303.243775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2303.280086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2303.301376] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2303.944668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2303.972556] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 2304.794347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2304.850331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2305.154489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2305.197957] LNetError: 116814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2305.205764] LNet: Removed LNI 192.168.204.2@tcp [ 2305.235269] LNet: Removed LNI 192.168.204.2@tcp2 [ 2305.271461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2305.391466] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2305.400742] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 2305.411245] LNet: 116820:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2305.858812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2306.107818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2307.172318] LNet: Removed LNI 192.168.204.2@tcp [ 2307.178250] LNetError: 116857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2307.194890] LNetError: 116857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2307.237601] LNet: Removed LNI 192.168.204.2@tcp2 [ 2307.273208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2307.383659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2307.413494] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2308.334247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2308.359410] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 2309.591904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2309.627240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2309.679667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2309.914925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2309.941834] LNet: 117142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.950973] LNet: 117142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2309.967964] LNetError: 117142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.978637] LNet: Removed LNI 192.168.204.2@tcp [ 2311.014874] LNet: Removed LNI 192.168.204.2@tcp2 [ 2311.093818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.221517] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2311.232291] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 2311.243701] LNet: 117148:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2312.085375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2312.327451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2312.375711] LNet: Removed LNI 192.168.204.2@tcp [ 2313.463284] LNet: Removed LNI 192.168.204.2@tcp2 [ 2313.533442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2313.664534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2313.713081] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2314.646107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2314.670536] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 2315.773562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2315.836125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2316.138431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2316.179937] LNetError: 117465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2316.188891] LNetError: 117465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2316.199652] LNet: Removed LNI 192.168.204.2@tcp [ 2316.225499] LNet: Removed LNI 192.168.204.2@tcp2 [ 2316.290832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.432809] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2316.437798] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 2316.443156] LNet: 117471:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2316.980466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2317.234393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2318.303170] LNet: Removed LNI 192.168.204.2@tcp [ 2318.316189] LNet: Removed LNI 192.168.204.2@tcp2 [ 2318.366304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2318.446037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2318.489244] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2319.561786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2319.583357] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 2320.664748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2320.739394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2320.835480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2321.095627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2321.129487] LNet: 117793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2321.140582] LNet: 117793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2322.207166] LNet: Removed LNI 192.168.204.2@tcp [ 2323.240691] LNet: Removed LNI 192.168.204.2@tcp2 [ 2323.288075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.428841] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2323.439745] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 2323.451909] LNet: 117799:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2324.295429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2324.539127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2324.581450] LNetError: 117836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.592791] LNetError: 117836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2324.616959] LNet: Removed LNI 192.168.204.2@tcp [ 2325.679519] LNet: Removed LNI 192.168.204.2@tcp2 [ 2325.728601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2325.812589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2325.853035] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2326.459287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2326.478717] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 2327.451373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2327.484238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2327.578394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2328.095349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2329.184257] LNet: Removed LNI 192.168.204.2@tcp [ 2329.214552] LNet: Removed LNI 192.168.204.2@tcp2 [ 2329.245544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2329.352333] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2329.357232] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 2329.362255] LNet: 118139:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2329.948215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2330.455687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2331.551338] LNet: Removed LNI 192.168.204.2@tcp [ 2331.565836] LNet: Removed LNI 192.168.204.2@tcp2 [ 2331.600283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.690174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2331.725436] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2332.349475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2332.370459] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 2333.456911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2333.544254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2333.578686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2333.659416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2334.101640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.135240] LNet: Removed LNI 192.168.204.2@tcp [ 2336.166155] LNet: Removed LNI 192.168.204.2@tcp2 [ 2336.197392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.286265] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2336.305835] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 2337.352097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2345.448528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2347.427670] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:40:37 (1788972037) [ 2347.539270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2347.573953] LNet: 119254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2347.580756] LNet: 119254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2347.589568] LNetError: 119254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2347.596965] LNetError: 119254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2347.604918] LNet: Removed LNI 192.168.204.2@tcp [ 2348.648700] LNet: Removed LNI 192.168.204.2@tcp2 [ 2348.698445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2349.125623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2349.153584] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2349.958895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2349.988901] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2351.314287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2351.346621] LNet: Removed LNI 10.1.2.3@tcp [ 2351.356631] LNet: Removed LNI 192.168.204.2@tcp [ 2351.395960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2351.544476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2351.613638] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2351.622923] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2352.404043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2352.432808] LNet: Removed LNI 192.168.204.2@tcp [ 2352.448939] LNet: Removed LNI 10.1.2.3@tcp [ 2360.136022] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 12:40:49 (1788972049) [ 2360.313941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2360.385910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.503765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2360.546321] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2361.442603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2361.480391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2361.523574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2361.570982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2361.617679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2361.664367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2361.700130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2361.732772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2361.763262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2361.797089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2361.842942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2361.883617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2362.043701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2369.629308] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 12:40:59 (1788972059) [ 2369.734844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2370.867426] LNet: Removed LNI 192.168.204.2@tcp [ 2370.928432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.023369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2371.049980] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2371.884105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2371.928222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2371.965442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.008227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.048490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.111265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.144339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.195456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.239438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.297962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.332503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.367423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.446651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2372.476584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.506472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.547254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.599186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.650614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.708601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.784160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.861833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.917439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2372.960446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.2@tcp [ 2379.615815] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:41:09 (1788972069) [ 2379.698697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2379.722095] LNet: 121624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2379.729890] LNet: 121624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2379.751908] LNetError: 121624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2379.759884] LNetError: 121624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2379.785457] LNet: Removed LNI 192.168.204.2@tcp [ 2379.843318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2379.963178] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2380.088280] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2380.091222] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2380.094181] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2382.468039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2382.543739] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2383.376316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2383.397828] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2385.472480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2385.532617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2385.605895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.2@tcp -m PUT -e local_timeout -r 1 [ 2385.646301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2385.716888] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2385.719977] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2386.150769] LNetError: 121698:0:(lib-move.c:4794:lnet_parse()) 192.168.204.2@tcp, src 192.168.204.2@tcp: Dropping PUT (error -108 looking up sender) [ 2387.176637] LNet: Removed LNI 10.1.2.3@tcp [ 2387.206577] LNet: Removed LNI 192.168.204.2@tcp [ 2387.847993] Key type .llcrypt unregistered [ 2387.857421] Key type ._llcrypt unregistered [ 2401.045746] Key type ._llcrypt registered [ 2401.049395] Key type .llcrypt registered [ 2402.910695] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:41:32 (1788972092) [ 2403.217451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2403.234871] alg: No test for adler32 (adler32-zlib) [ 2404.125306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2404.385142] LNet: 123007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2405.015384] Key type .llcrypt unregistered [ 2405.020728] Key type ._llcrypt unregistered [ 2411.474925] Key type ._llcrypt registered [ 2411.480415] Key type .llcrypt registered [ 2411.820121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2411.845844] alg: No test for adler32 (adler32-zlib) [ 2412.764770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2412.791315] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2420.410166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.102@tcp [ 2420.588830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2420.970212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.102@tcp [ 2421.028150] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.102@tcp [ 2421.233625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2421.257876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2421.276948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2421.298976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2421.325962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2421.359699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2421.414732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2421.473955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2421.526830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2421.558390] LNet: 123456:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2421.573993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2421.619428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2421.663799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2421.699411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2421.786490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2421.816650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2422.375976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2422.621548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2422.662539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2422.697561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2422.727902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2422.778526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2422.855166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2422.906516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2422.943108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2422.969971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2422.986458] LNet: 123575:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2423.004435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2423.026457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2423.047658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2423.077619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2423.119512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2423.161574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2423.788660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.102@tcp [ 2424.005633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2424.028648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2424.058983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2424.088996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2424.113594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2424.141240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2424.167362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2424.209670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2424.244327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2424.268301] LNet: 123695:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2424.292488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2424.337712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2424.378300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2424.406266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2424.444734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2424.490536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2424.969635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2425.221207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2425.252918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2425.284371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2425.313370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2425.343767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2425.384720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2425.425782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2425.499238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2425.543274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2425.582262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2425.613639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2425.646547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2425.678343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2425.706714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2425.742712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2426.080631] LNet: 123860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2426.092604] LNetError: 123860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.108186] LNet: Removed LNI 192.168.204.2@tcp [ 2426.426583] Key type .llcrypt unregistered [ 2426.430590] Key type ._llcrypt unregistered [ 2442.086857] Key type ._llcrypt registered [ 2442.089804] Key type .llcrypt registered [ 2443.572590] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:42:13 (1788972133) [ 2444.045627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2444.071536] alg: No test for adler32 (adler32-zlib) [ 2444.973322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2445.463742] LNet: 124745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2446.016445] Key type .llcrypt unregistered [ 2446.019140] Key type ._llcrypt unregistered [ 2452.726865] Key type ._llcrypt registered [ 2452.732173] Key type .llcrypt registered [ 2453.122489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2453.179204] alg: No test for adler32 (adler32-zlib) [ 2454.006465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2454.022642] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2462.595950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.102@tcp [ 2473.848344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2473.890290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2473.910370] LNet: Added LNI 192.168.204.2@tcp1 [8/256/0/180] [ 2475.185625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2476.250312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.102@tcp [ 2476.576167] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.102@tcp [ 2476.764487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2476.812490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2476.858899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2476.908989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2476.964875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2477.004493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2477.066772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2477.121745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2477.213611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2477.244834] LNet: 125398:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2477.280136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2477.345899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2477.394739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2477.456888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2477.505038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2477.573959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2478.705127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2479.307593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2479.398548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2479.495537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2479.567443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2479.663909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2479.737411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2479.791348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2479.828254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2479.868651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2479.896071] LNet: 125518:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2479.913436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2479.978699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2480.014995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2480.063251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2480.127247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2480.167924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2480.967613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.102@tcp [ 2481.179077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2481.215116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2481.237934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2481.260857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2481.285486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2481.314322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2481.343117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2481.379398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2481.412829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2481.470887] LNet: 125637:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2481.503330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2481.537021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2481.565312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2481.595872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2481.642966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2481.684248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2482.690839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2483.108815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2483.138419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2483.188953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2483.232660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2483.255496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2483.281297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2483.329691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2483.357847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2483.432481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2483.488573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2483.559522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2483.603658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2483.647665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2483.699955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2483.787082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2490.862434] LNet: 125838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2490.869330] LNetError: 125838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2490.879669] LNet: Removed LNI 192.168.204.2@tcp [ 2490.891912] LNet: Removed LNI 192.168.204.2@tcp1 [ 2491.263125] Key type .llcrypt unregistered [ 2491.268872] Key type ._llcrypt unregistered [ 2511.571889] Key type ._llcrypt registered [ 2511.574491] Key type .llcrypt registered [ 2512.992334] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 12:43:22 (1788972202) [ 2513.341559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2513.352875] alg: No test for adler32 (adler32-zlib) [ 2514.289548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2514.375396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2514.415659] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2515.157284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2515.196819] LNet: Added LNI 192.168.204.2@tcp2 [8/256/0/180] [ 2516.210407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.2@tcp [ 2516.325628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2517.407192] LNet: Removed LNI 192.168.204.2@tcp [ 2523.967844] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 12:43:34 (1788972214) [ 2525.722326] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2527.257326] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:43:37 (1788972217) [ 2534.313416] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:43:44 (1788972224) [ 2542.573097] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:43:52 (1788972232) [ 2550.363765] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:44:00 (1788972240) [ 2557.542703] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:44:07 (1788972247) [ 2564.845763] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:44:14 (1788972254) [ 2573.975813] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:44:23 (1788972263) [ 2582.243456] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:44:31 (1788972271) [ 2591.001774] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:44:40 (1788972280) [ 2598.650414] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:44:48 (1788972288) [ 2608.005781] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:44:57 (1788972297) [ 2615.890849] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:45:05 (1788972305) [ 2623.478910] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:45:13 (1788972313) [ 2631.684407] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:45:21 (1788972321) [ 2640.141763] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:45:29 (1788972329) [ 2647.981741] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:45:37 (1788972337) [ 2656.006498] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:45:45 (1788972345) [ 2664.182723] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:45:53 (1788972353) [ 2664.282250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2664.319757] LNet: 137529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2664.338438] LNetError: 137529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2664.354214] LNet: Removed LNI 192.168.204.2@tcp2 [ 2664.381992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2664.539644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2664.587671] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2666.054279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2666.075990] LNet: 137677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2666.091643] LNetError: 137677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2666.112833] LNet: Removed LNI 192.168.204.2@tcp [ 2666.154424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2666.202990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2666.231061] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2673.773613] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:46:03 (1788972363) [ 2681.795578] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:46:11 (1788972371) [ 2688.322205] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:46:18 (1788972378) [ 2695.201264] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:46:25 (1788972385) [ 2701.685627] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:46:31 (1788972391) [ 2708.496976] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:46:38 (1788972398) [ 2715.766442] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:46:45 (1788972405) [ 2724.108415] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:46:53 (1788972413) [ 2733.175645] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:47:03 (1788972423) [ 2741.260320] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:47:11 (1788972431) [ 2748.553189] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:47:18 (1788972438) [ 2756.067442] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:47:26 (1788972446) [ 2762.739210] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2495 sec ========== 12:47:32 (1788972452) [ 2763.011804] LNet: 145144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2763.027773] LNetError: 145144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2763.061322] LNet: Removed LNI 192.168.204.2@tcp [ 2763.742809] Key type .llcrypt unregistered [ 2763.745872] Key type ._llcrypt unregistered [ 2775.429923] Key type ._llcrypt registered [ 2775.433156] Key type .llcrypt registered [ 2775.823050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2775.857812] alg: No test for adler32 (adler32-zlib) [ 2776.987117] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 2777.328374] LNet: Added LNI 192.168.204.2@tcp [8/256/0/180] [ 2779.055469] Key type lgssc registered [ 2780.266835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2845.328302] Lustre: Mounted lustre-client - version 2.17.58_42_gf11d80b [ 2850.623068] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2858.657503] Lustre: Unmounted lustre-client [ 2899.254776] Key type lgssc unregistered [ 2899.523789] LNet: 148092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2899.534314] LNetError: 148092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2899.558993] LNet: Removed LNI 192.168.204.2@tcp [ 2900.518588] Key type .llcrypt unregistered [ 2900.523318] Key type ._llcrypt unregistered