[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 440037220 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002375] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007408] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.012033] SELinux: Initializing. [ 0.013057] *** VALIDATE selinux *** [ 0.020705] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024752] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026103] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027083] *** VALIDATE tmpfs *** [ 0.028330] *** VALIDATE proc *** [ 0.029222] *** VALIDATE cgroup *** [ 0.030008] *** VALIDATE cgroup2 *** [ 0.032074] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033118] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035030] Spectre V2 : User space: Vulnerable [ 0.036008] Speculative Store Bypass: Vulnerable [ 0.038878] debug: unmapping init [mem 0xffffffff85459000-0xffffffff85460fff] [ 0.040782] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041551] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042018] ... version: 2 [ 0.042952] ... bit width: 48 [ 0.043013] ... generic registers: 4 [ 0.044011] ... value mask: 0000ffffffffffff [ 0.045011] ... max period: 00007fffffffffff [ 0.046014] ... fixed-purpose events: 3 [ 0.047011] ... event mask: 000000070000000f [ 0.049219] rcu: Hierarchical SRCU implementation. [ 0.051314] smp: Bringing up secondary CPUs ... [ 0.052506] x86: Booting SMP configuration: [ 0.053024] .... node #0, CPUs: #1 #2 #3 [ 0.056478] smp: Brought up 1 node, 4 CPUs [ 0.058011] smpboot: Max logical packages: 1 [ 0.059024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.204030] node 0 deferred pages initialised in 139ms [ 0.208023] devtmpfs: initialized [ 0.209287] x86/mm: Memory block size: 128MB [ 0.212106] gcov: version magic: 0x41383552 [ 0.216347] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.217072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.218251] pinctrl core: initialized pinctrl subsystem [ 0.219191] [ 0.219700] ************************************************************* [ 0.220013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.221014] ** ** [ 0.222014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.223016] ** ** [ 0.224016] ** This means that this kernel is built to expose internal ** [ 0.225016] ** IOMMU data structures, which may compromise security on ** [ 0.226015] ** your system. ** [ 0.227015] ** ** [ 0.228016] ** If you see this message and you are not debugging the ** [ 0.229016] ** kernel, report this immediately to your vendor! ** [ 0.230016] ** ** [ 0.231017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.232017] ************************************************************* [ 0.234000] NET: Registered protocol family 16 [ 0.235406] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.238057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.240070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.243040] cpuidle: using governor menu [ 0.244902] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.247601] PCI: Using configuration type 1 for base access [ 0.250151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.260057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.262056] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.266041] cryptd: max_cpu_qlen set to 1000 [ 0.269281] ACPI: Added _OSI(Module Device) [ 0.271017] ACPI: Added _OSI(Processor Device) [ 0.272018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.273020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.277987] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.284755] ACPI: Interpreter enabled [ 0.286098] ACPI: PM: (supports S0 S3 S4 S5) [ 0.288017] ACPI: Using IOAPIC for interrupt routing [ 0.290233] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.293456] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.303436] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.306071] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.309029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.313105] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.318631] acpiphp: Slot [2] registered [ 0.321343] acpiphp: Slot [5] registered [ 0.324138] acpiphp: Slot [6] registered [ 0.325199] acpiphp: Slot [3] registered [ 0.327118] acpiphp: Slot [4] registered [ 0.328098] acpiphp: Slot [7] registered [ 0.329106] acpiphp: Slot [8] registered [ 0.331144] acpiphp: Slot [9] registered [ 0.332098] acpiphp: Slot [10] registered [ 0.333122] acpiphp: Slot [11] registered [ 0.335113] acpiphp: Slot [12] registered [ 0.336118] acpiphp: Slot [13] registered [ 0.338098] acpiphp: Slot [14] registered [ 0.339107] acpiphp: Slot [15] registered [ 0.341191] acpiphp: Slot [16] registered [ 0.343127] acpiphp: Slot [17] registered [ 0.344123] acpiphp: Slot [18] registered [ 0.346115] acpiphp: Slot [19] registered [ 0.347115] acpiphp: Slot [20] registered [ 0.349129] acpiphp: Slot [21] registered [ 0.350142] acpiphp: Slot [22] registered [ 0.351113] acpiphp: Slot [23] registered [ 0.353118] acpiphp: Slot [24] registered [ 0.354108] acpiphp: Slot [25] registered [ 0.355143] acpiphp: Slot [26] registered [ 0.357117] acpiphp: Slot [27] registered [ 0.358118] acpiphp: Slot [28] registered [ 0.359139] acpiphp: Slot [29] registered [ 0.361129] acpiphp: Slot [30] registered [ 0.363142] acpiphp: Slot [31] registered [ 0.364099] PCI host bridge to bus 0000:00 [ 0.365024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.368042] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.370033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.372032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.375035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.378036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.380204] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.383177] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.386414] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.393938] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.398078] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.401034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.403020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.405026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.407725] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.410914] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.413062] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.415758] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.420019] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.430735] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.435019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.440048] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.447019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.452019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.463021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.472342] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.480018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.486022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.502027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.512846] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.515432] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.518395] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.522427] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.525361] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.529268] iommu: Default domain type: Passthrough [ 0.531542] SCSI subsystem initialized [ 0.533141] ACPI: bus type USB registered [ 0.534216] usbcore: registered new interface driver usbfs [ 0.536118] usbcore: registered new interface driver hub [ 0.537105] usbcore: registered new device driver usb [ 0.539205] pps_core: LinuxPPS API ver. 1 registered [ 0.541017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.544090] PTP clock support registered [ 0.546074] EDAC MC: Ver: 3.0.0 [ 0.548136] PCI: Using ACPI for IRQ routing [ 0.550940] NetLabel: Initializing [ 0.552017] NetLabel: domain hash size = 128 [ 0.554016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.556091] NetLabel: unlabeled traffic allowed by default [ 0.559145] vgaarb: loaded [ 0.561300] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.564016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.573000] clocksource: Switched to clocksource kvm-clock [ 0.695924] VFS: Disk quotas dquot_6.6.0 [ 0.697271] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.699917] *** VALIDATE ramfs *** [ 0.701283] *** VALIDATE hugetlbfs *** [ 0.703188] pnp: PnP ACPI init [ 0.706828] pnp: PnP ACPI: found 6 devices [ 0.724292] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.728131] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.730354] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.732765] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.735018] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.737215] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.740056] NET: Registered protocol family 2 [ 0.743245] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.748181] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.751745] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.756750] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.760222] TCP: Hash tables configured (established 65536 bind 65536) [ 0.762913] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.765664] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.768089] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.770671] NET: Registered protocol family 1 [ 0.773446] RPC: Registered named UNIX socket transport module. [ 0.775572] RPC: Registered udp transport module. [ 0.777359] RPC: Registered tcp transport module. [ 0.779324] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.781614] NET: Registered protocol family 44 [ 0.783264] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.785352] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.787563] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.790170] PCI: CLS 0 bytes, default 64 [ 0.792163] Unpacking initramfs... [ 2.223069] debug: unmapping init [mem 0xffffa01dfcc64000-0xffffa01dfffcffff] [ 2.226835] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.229332] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.231574] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.757297] Initialise system trusted keyrings [ 2.759497] Key type blacklist registered [ 2.761723] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.771347] zbud: loaded [ 2.774563] *** VALIDATE nfs *** [ 2.775818] *** VALIDATE nfs4 *** [ 2.777606] pstore: using deflate compression [ 2.783116] Platform Keyring initialized [ 2.898728] NET: Registered protocol family 38 [ 2.900810] Key type asymmetric registered [ 2.902477] Asymmetric key parser 'x509' registered [ 2.905354] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.908564] io scheduler mq-deadline registered [ 2.910274] io scheduler kyber registered [ 2.912317] io scheduler bfq registered [ 2.914214] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.917573] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.920848] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.924128] ACPI: Power Button [PWRF] [ 2.930394] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.937574] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.949258] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.982366] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.011858] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.017181] Non-volatile memory driver v1.3 [ 3.018975] Linux agpgart interface v0.103 [ 3.052402] virtio_blk virtio1: [vda] 146864 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.056844] vda: detected capacity change from 0 to 75194368 [ 3.073995] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.077275] vdb: detected capacity change from 0 to 1073741824 [ 3.087415] libphy: Fixed MDIO Bus: probed [ 3.093957] usbcore: registered new interface driver usbserial_generic [ 3.096268] usbserial: USB Serial support registered for generic [ 3.099023] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.103081] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.105102] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.108130] mousedev: PS/2 mouse device common for all mice [ 3.111719] rtc_cmos 00:05: RTC can wake from S4 [ 3.118089] rtc_cmos 00:05: registered as rtc0 [ 3.118746] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.120401] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.120453] intel_pstate: CPU model not supported [ 3.128354] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.135774] hid: raw HID events driver (C) Jiri Kosina [ 3.136311] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.138158] usbcore: registered new interface driver usbhid [ 3.143801] usbhid: USB HID core driver [ 3.146134] drop_monitor: Initializing network drop monitor service [ 3.151609] Initializing XFRM netlink socket [ 3.154960] NET: Registered protocol family 10 [ 3.158511] Segment Routing with IPv6 [ 3.160125] NET: Registered protocol family 17 [ 3.161972] mpls_gso: MPLS GSO support [ 3.169628] RAS: Correctable Errors collector initialized. [ 3.172161] AVX version of gcm_enc/dec engaged. [ 3.173544] AES CTR mode by8 optimization enabled [ 3.247290] sched_clock: Marking stable (3247269359, 0)->(4137304290, -890034931) [ 3.251063] registered taskstats version 1 [ 3.252924] Loading compiled-in X.509 certificates [ 3.254726] zswap: loaded using pool lzo/zbud [ 3.277747] Key type big_key registered [ 3.290950] Key type encrypted registered [ 3.293040] ima: No TPM chip found, activating TPM-bypass! [ 3.294978] ima: Allocated hash algorithm: sha1 [ 3.296562] ima: No architecture policies found [ 3.298293] evm: Initialising EVM extended attributes: [ 3.300118] evm: security.selinux [ 3.301346] evm: security.ima [ 3.302336] evm: security.capability [ 3.303876] evm: HMAC attrs: 0x1 [ 3.306152] rtc_cmos 00:05: setting system clock to 2026-08-27 16:11:44 UTC (1787847104) [ 3.312671] debug: unmapping init [mem 0xffffffff86403000-0xffffffff865fffff] [ 3.316163] debug: unmapping init [mem 0xffffffff85182000-0xffffffff85458fff] [ 3.327099] Write protecting the kernel read-only data: 28672k [ 3.330302] debug: unmapping init [mem 0xffffffff83803000-0xffffffff839fffff] [ 3.333241] debug: unmapping init [mem 0xffffffff84114000-0xffffffff841fffff] [ 3.363332] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.372078] systemd[1]: Detected virtualization kvm. [ 3.374234] systemd[1]: Detected architecture x86-64. [ 3.376050] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.402095] systemd[1]: No hostname configured. [ 3.403728] systemd[1]: Set hostname to . [ 3.405527] random: systemd: uninitialized urandom read (16 bytes read) [ 3.407512] systemd[1]: Initializing machine ID from random generator. [ 3.528277] random: systemd: uninitialized urandom read (16 bytes read) [ 3.532855] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.537776] random: systemd: uninitialized urandom read (16 bytes read) [ 3.540169] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.544722] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.095849] device-mapper: uevent: version 1.0.3 [ 4.098416] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 4.831925] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.904905] virtio_net virtio0 ens2: renamed from eth0 [ 4.954461] scsi host0: ata_piix [ 5.029204] scsi host1: ata_piix [ 5.030658] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.032925] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.190955] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.813651] random: crng init done [ 9.815184] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.432971] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.300058] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.964092] SELinux: Disabled at runtime. [ 15.061442] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.073275] systemd[1]: Detected virtualization kvm. [ 15.075742] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.274693] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.291678] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.310755] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.328493] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.338747] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.363511] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.376146] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped targ[ 17.045056] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS et Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 17.948478] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.835646] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.836671] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.372214] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.451053] EDAC sbridge: Ver: 1.1.2 [ 21.861415] Key type dns_resolver registered [ 22.391128] NFS: Registering the id_resolver key type [ 22.396303] Key type id_resolver registered [ 22.407451] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg610-client login: [ 95.402450] libcfs: loading out-of-tree module taints kernel. [ 95.458138] Key type ._llcrypt registered [ 95.463960] Key type .llcrypt registered [ 95.948511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 95.969617] alg: No test for adler32 (adler32-zlib) [ 97.323371] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 98.215358] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 99.967162] Key type lgssc registered [ 101.628081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 223.979783] Lustre: Mounted lustre-client - version 2.17.57_87_ge30c52f [ 228.327766] hrtimer: interrupt took 3860947 ns [ 228.643344] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.137236] Lustre: DEBUG MARKER: oleg610-client.virtnet: executing check_logdir /tmp/testlogs/ [ 248.655220] Lustre: DEBUG MARKER: oleg610-client.virtnet: executing yml_node [ 249.839196] Lustre: lustre-OST0000-osc-ffffa01e5121b000: disconnect after 24s idle [ 252.530482] Lustre: DEBUG MARKER: Client: 2.17.57.87 [ 255.034461] Lustre: DEBUG MARKER: MDS: 2.17.57.87 [ 257.314467] Lustre: DEBUG MARKER: OSS: 2.17.57.87 [ 258.866737] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Aug 27 12:15:58 EDT 2026 [ 273.956543] Lustre: Unmounted lustre-client [ 289.829559] Key type lgssc unregistered [ 290.108935] LNet: 8432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.118194] LNetError: 8432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 290.144503] LNet: Removed LNI 192.168.206.10@tcp [ 291.089340] Key type .llcrypt unregistered [ 291.092325] Key type ._llcrypt unregistered [ 314.426922] Key type ._llcrypt registered [ 314.429526] Key type .llcrypt registered [ 314.651255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 314.676186] alg: No test for adler32 (adler32-zlib) [ 315.591904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 315.635964] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 315.647678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 319.192842] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 319.295390] LNet: 9316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.306899] LNetError: 9316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 319.344665] LNet: Removed LNI 192.168.206.10@tcp [ 320.037501] Key type .llcrypt unregistered [ 320.041630] Key type ._llcrypt unregistered [ 325.612233] Key type ._llcrypt registered [ 325.613880] Key type .llcrypt registered [ 326.712510] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:17:06 (1787847426) [ 326.935983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 326.947921] alg: No test for adler32 (adler32-zlib) [ 327.813954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.940064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.966926] LNet: 9734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.004457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.074538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 328.089882] LNet: 9741:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 334.911506] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:17:14 (1787847434) [ 335.011093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.076941] LNet: 10312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.130237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.263776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 343.172404] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:17:22 (1787847442) [ 343.297098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.338063] LNet: 10900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.374565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.506431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 350.202812] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:17:30 (1787847450) [ 350.435876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.498332] LNet: 11485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.536042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.662774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 356.545968] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:17:36 (1787847456) [ 356.637334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.662201] LNet: 12070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.691033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.798746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 357.214593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 363.224757] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:17:43 (1787847463) [ 363.321476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.354137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.408433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 363.638474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 364.245123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 371.891436] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:17:51 (1787847471) [ 371.987933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.041184] LNet: 13256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.049734] LNet: 13256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 372.068520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.148618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 378.971331] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:17:58 (1787847478) [ 379.235546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.273299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.337223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 379.367487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 379.804790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 379.843620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 380.367193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 380.405279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 380.883660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 380.945621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 381.445569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 381.498692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 381.996738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 382.037073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 382.534335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 382.564683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 383.085665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 383.128477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 383.668968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 383.705766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 384.209260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 384.245089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 384.619829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 384.666323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 385.166305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 385.240148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 385.766138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 385.825459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 392.335556] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:18:12 (1787847492) [ 392.555607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.594826] LNet: 14530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.602461] LNet: 14530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 392.640114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.734842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 392.770869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 399.925375] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:18:19 (1787847499) [ 400.029721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.092587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.200853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 400.241811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 406.913616] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:18:26 (1787847506) [ 407.144211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.204921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.331530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 407.371164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 414.141984] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:18:33 (1787847513) [ 414.216869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.285087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.400899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 414.429914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 421.112416] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:18:40 (1787847520) [ 421.202075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.257960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.348677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 421.405669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 428.893340] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:18:48 (1787847528) [ 429.016952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.048825] LNet: 17457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 429.054683] LNet: 17457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 429.100532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.223849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 429.286569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 436.014935] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:18:55 (1787847535) [ 436.270818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.332042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.399321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 436.823807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 443.627415] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:19:03 (1787847543) [ 443.747527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.823020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.083499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 444.339867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 450.147759] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:19:10 (1787847550) [ 450.462837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.526594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.660750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 450.881837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 457.183956] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:19:17 (1787847557) [ 457.488878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.549874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.981073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 458.345744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 464.386840] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:19:24 (1787847564) [ 464.493915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.554594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.647827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 464.724850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 471.246294] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:19:31 (1787847571) [ 471.355880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.397196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.487805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 471.529636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 477.350339] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:19:37 (1787847577) [ 477.462700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.517834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.618756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 484.079855] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:19:44 (1787847584) [ 484.190558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.260704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.397742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 491.089970] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:19:50 (1787847590) [ 491.260650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.311738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.403598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 491.850909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 498.177126] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:19:58 (1787847598) [ 498.501497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.553694] LNet: 23488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 498.563971] LNet: 23488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 498.596984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.686487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 498.725402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 505.753281] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:20:05 (1787847605) [ 505.853410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.968617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.112715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 506.177679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 513.394501] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:20:13 (1787847613) [ 513.530835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.596331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.698432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 513.752790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 521.131569] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:20:20 (1787847620) [ 521.362916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.391473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.482239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 521.529632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 528.798832] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:20:28 (1787847628) [ 528.893535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.949322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.029900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 529.118299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 529.135294] LNetError: 25846:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 529.172538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 529.210818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 535.854754] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:20:35 (1787847635) [ 536.095057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.164495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.255272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Lg2hTH7BcM [ 536.355311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Lg2hTH7BcM [ 536.408429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Lg2hTH7BcM [ 542.838234] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:20:42 (1787847642) [ 543.159035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.184465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.275632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 543.328617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 543.384705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 549.386821] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 12:20:49 (1787847649) [ 549.571034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.603762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.677225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 549.706554] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 551.046438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 552.367911] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 554.304627] Key type lgssc registered [ 555.462190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 558.982419] LNetError: 27607:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.10@tcp99 has no available nets [ 561.730853] Key type lgssc unregistered [ 562.106207] LNetError: 28719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 562.122660] LNet: Removed LNI 192.168.206.10@tcp [ 562.659428] Key type .llcrypt unregistered [ 562.661892] Key type ._llcrypt unregistered [ 569.416404] Key type ._llcrypt registered [ 569.418606] Key type .llcrypt registered [ 570.915641] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:21:10 (1787847670) [ 571.252314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 571.273393] alg: No test for adler32 (adler32-zlib) [ 572.163389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.231991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 575.221122] LNet: Message forwarding will be enabled in 2s [ 578.335108] LNet: Message forwarding enabled [ 578.749707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 578.788111] LNet: Message forwarding will stop in 2s to 6s [ 583.327869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 584.193281] LNet: Message forwarding disabled [ 586.164001] LNet: Message forwarding will be enabled in 2s [ 589.345905] LNet: Message forwarding enabled [ 594.142826] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:21:34 (1787847694) [ 594.430142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.466989] LNet: 30128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.117728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.197578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 602.562748] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:21:42 (1787847702) [ 602.664918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.696910] LNet: 30703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.751290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.820878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 602.878148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 602.922961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 602.952217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 602.981320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 603.002494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 603.024981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 603.049681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 603.081820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 603.115900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 603.166270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 603.205852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 603.247878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 603.285151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 603.310843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 603.344599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 603.387907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 603.412872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 603.452214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 603.465614] LNetError: 30753:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 609.283851] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:21:49 (1787847709) [ 609.550671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.587918] LNet: 31324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.611442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.700766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 615.654356] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:21:55 (1787847715) [ 617.300210] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg610-server [ 619.142915] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:21:58 (1787847718) [ 619.203101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.217854] LNet: 32248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 619.232509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.285798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 619.305724] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 620.104707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[11-15]@tcp [ 620.127948] LNet: 32387:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 620.164943] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.11@tcp is being used as a gateway but routing feature is not turned on [ 620.174993] LNetError: 32257:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.11@tcp has gone from up to down [ 621.166648] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.11@tcp is being used as a gateway but routing feature is not turned on [ 621.183421] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 622.185740] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.15@tcp is being used as a gateway but routing feature is not turned on [ 622.198499] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 624.235868] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.11@tcp is being used as a gateway but routing feature is not turned on [ 624.255369] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 626.261659] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:22:06 (1787847726) [ 628.052740] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg610-server [ 628.324355] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.15@tcp is being used as a gateway but routing feature is not turned on [ 628.334584] LNetError: 32257:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 629.813236] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:22:09 (1787847729) [ 629.959714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.994920] LNet: 33321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.999412] LNetError: 33321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.024513] LNet: Removed LNI 192.168.206.10@tcp [ 630.085485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.140406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 630.159987] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 630.863913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[11-16/2]@tcp [ 630.904847] LNet: 33440:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 630.916624] LNet: 33440:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 630.943805] LNetError: 33327:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.13@tcp has gone from up to down [ 630.963503] LNetError: 33327:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 630.966742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[11-16/2]@tcp [ 637.632208] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:22:17 (1787847737) [ 637.877774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.899864] LNetError: 34032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.924248] LNet: Removed LNI 192.168.206.10@tcp [ 637.989034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.125770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 638.174991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.192458] LNet: 34046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.201503] LNet: 34046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 638.227095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.324484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.391885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.431879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.484515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 638.539638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.588052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.653382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.719273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.749430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.799846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 638.865666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.915582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.997644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 639.102298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.177662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.230488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 639.302119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.339791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.400859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 639.475632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.519298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.569911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 639.652795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.701505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.787386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 639.857192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.894091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.944775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 639.990515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.036965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.101361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.341296] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:22:25 (1787847745) [ 646.965472] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg610-server [ 648.525935] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:22:28 (1787847748) [ 650.032181] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg610-server [ 651.625937] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:22:31 (1787847751) [ 652.487178] Key type .llcrypt unregistered [ 652.489557] Key type ._llcrypt unregistered [ 658.679713] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 658.805113] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 658.814339] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 658.820465] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 661.365500] Key type ._llcrypt registered [ 661.367585] Key type .llcrypt registered [ 661.479771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 661.488579] alg: No test for adler32 (adler32-zlib) [ 662.331567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.401446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 662.432652] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 663.133974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 663.167906] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 664.984212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 665.000309] LNet: Removed LNI 10.1.2.3@tcp [ 665.132660] LNet: 36157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.145553] LNetError: 36157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.182322] LNet: Removed LNI 192.168.206.10@tcp [ 665.719725] Key type .llcrypt unregistered [ 665.725056] Key type ._llcrypt unregistered [ 676.144324] Key type ._llcrypt registered [ 676.146242] Key type .llcrypt registered [ 677.322505] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:22:57 (1787847777) [ 677.901538] Key type .llcrypt unregistered [ 677.907285] Key type ._llcrypt unregistered [ 684.411736] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 684.579864] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 684.594707] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 684.612993] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 687.202968] Key type ._llcrypt registered [ 687.205915] Key type .llcrypt registered [ 687.369920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 687.379215] alg: No test for adler32 (adler32-zlib) [ 688.323414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.372775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 688.398530] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 689.052465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 689.111088] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 691.547256] LNet: 37662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.555614] LNetError: 37662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 691.561374] LNet: Removed LNI 10.1.2.3@tcp [ 691.576465] LNet: Removed LNI 192.168.206.10@tcp [ 692.113544] Key type .llcrypt unregistered [ 692.116736] Key type ._llcrypt unregistered [ 703.015627] Key type ._llcrypt registered [ 703.017332] Key type .llcrypt registered [ 704.427615] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:23:23 (1787847803) [ 704.940531] Key type .llcrypt unregistered [ 704.943500] Key type ._llcrypt unregistered [ 711.304965] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 711.405174] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 711.417505] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 711.428303] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 714.077029] Key type ._llcrypt registered [ 714.078671] Key type .llcrypt registered [ 714.313845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 714.330632] alg: No test for adler32 (adler32-zlib) [ 715.278794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.320359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 715.345828] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 716.979474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 717.011788] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 718.691259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 718.720311] LNet: Removed LNI 10.1.2.3@tcp [ 718.740615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 718.782583] LNet: Removed LNI 10.1.2.31@tcp [ 718.798862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 718.833795] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 719.414658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 719.452057] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 720.526057] LNet: 39585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.535129] LNetError: 39585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 720.541499] LNet: Removed LNI 10.1.2.31@tcp [ 720.565573] LNet: Removed LNI 10.1.2.3@tcp [ 721.156552] Key type .llcrypt unregistered [ 721.162091] Key type ._llcrypt unregistered [ 732.524323] Key type ._llcrypt registered [ 732.526848] Key type .llcrypt registered [ 733.738941] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:23:53 (1787847833) [ 734.240574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 734.267734] alg: No test for adler32 (adler32-zlib) [ 735.143321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.218941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 735.243622] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 736.018775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.041898] LNet: 40541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.048772] LNetError: 40541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.075036] LNet: Removed LNI 192.168.206.10@tcp [ 736.100630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.163633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.10@tcp [ 736.184994] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 736.863403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.10@tcp [ 736.897487] LNet: Removed LNI 192.168.206.10@tcp [ 736.945442] LNet: 40676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.550560] Key type .llcrypt unregistered [ 737.552268] Key type ._llcrypt unregistered [ 747.132624] Key type ._llcrypt registered [ 747.134931] Key type .llcrypt registered [ 748.030524] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:24:08 (1787847848) [ 749.558170] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg610-server [ 751.024968] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:24:11 (1787847851) [ 751.542290] Key type .llcrypt unregistered [ 751.546540] Key type ._llcrypt unregistered [ 756.256709] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 756.416528] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 756.419505] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 756.423628] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 759.055763] Key type ._llcrypt registered [ 759.060240] Key type .llcrypt registered [ 759.253368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 759.268546] alg: No test for adler32 (adler32-zlib) [ 760.188290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.248662] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 760.908338] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 762.894975] LNet: 42520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.900895] LNetError: 42520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.910065] LNet: Removed LNI 192.168.206.10@tcp1 [ 762.922057] LNet: Removed LNI 10.1.2.3@tcp2 [ 763.543328] Key type .llcrypt unregistered [ 763.546963] Key type ._llcrypt unregistered [ 775.585398] Key type ._llcrypt registered [ 775.590727] Key type .llcrypt registered [ 776.816931] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:24:36 (1787847876) [ 777.187161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 777.200918] alg: No test for adler32 (adler32-zlib) [ 778.171881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.271536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.321786] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 779.106614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.129637] LNet: 43494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.138729] LNetError: 43494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.158404] LNet: Removed LNI 192.168.206.10@tcp [ 779.188806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.227963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 779.256093] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 784.763734] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:24:44 (1787847884) [ 784.930536] LNet: 44085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.939538] LNetError: 44085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.962579] LNet: Removed LNI 192.168.206.10@tcp [ 785.436769] Key type .llcrypt unregistered [ 785.439962] Key type ._llcrypt unregistered [ 791.657962] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 791.750507] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 791.756675] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 791.767647] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 794.395310] Key type ._llcrypt registered [ 794.397389] Key type .llcrypt registered [ 794.544837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 794.553235] alg: No test for adler32 (adler32-zlib) [ 795.441068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.482662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 795.495935] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 796.131269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 796.161696] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 798.057962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.085422] LNet: 44833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.098650] LNetError: 44833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 798.103194] LNet: Removed LNI 10.1.2.3@tcp [ 798.124411] LNet: Removed LNI 192.168.206.10@tcp [ 798.153518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.225270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 798.250454] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 798.256660] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 798.850790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.879320] LNet: 44860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.891138] LNetError: 44860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 798.895738] LNetError: 44860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 798.900528] LNet: Removed LNI 10.1.2.3@tcp [ 798.911341] LNet: Removed LNI 192.168.206.10@tcp [ 798.928753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.015452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 799.048978] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 799.058410] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 799.564760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 799.582829] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 800.541296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.557411] LNet: 45022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.566857] LNetError: 45022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 800.572923] LNetError: 45022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 800.577740] LNet: Removed LNI 10.1.2.3@tcp [ 800.579861] LNet: Removed LNI 192.168.206.10@tcp [ 800.590406] LNet: Removed LNI 192.168.206.10@tcp2 [ 800.624846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.686817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 800.720906] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 800.724071] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.727504] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 806.449780] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:25:06 (1787847906) [ 806.581791] LNet: 45611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.594710] LNetError: 45611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 806.604263] LNetError: 45611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 806.616098] LNet: Removed LNI 10.1.2.3@tcp [ 806.618899] LNet: Removed LNI 192.168.206.10@tcp [ 806.636542] LNet: Removed LNI 192.168.206.10@tcp2 [ 807.358981] Key type .llcrypt unregistered [ 807.362524] Key type ._llcrypt unregistered [ 813.292103] Key type ._llcrypt registered [ 813.295883] Key type .llcrypt registered [ 813.454353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 813.466109] alg: No test for adler32 (adler32-zlib) [ 814.405082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.490149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 814.544659] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 815.407287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.422386] LNet: 46018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.431637] LNetError: 46018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.454391] LNet: Removed LNI 192.168.206.10@tcp [ 815.486655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.555156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 815.588688] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 816.033675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.053756] LNet: 46048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.063589] LNetError: 46048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.076485] LNet: Removed LNI 192.168.206.10@tcp [ 816.108708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.181813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 816.204104] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 816.593693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.621291] LNet: Removed LNI 192.168.206.10@tcp [ 816.643873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.699276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 816.718938] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 817.200150] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 817.283129] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 817.287607] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 817.294974] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 819.687665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.711606] LNet: 46157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.718780] LNet: 46157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 819.725663] LNetError: 46157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.732157] LNetError: 46157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 819.745943] LNet: Removed LNI 192.168.206.10@tcp [ 819.764760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.820711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 819.866257] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 825.600843] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:25:25 (1787847925) [ 825.817042] LNet: 46754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.823669] LNetError: 46754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.843178] LNet: Removed LNI 192.168.206.10@tcp [ 826.605867] Key type .llcrypt unregistered [ 826.610341] Key type ._llcrypt unregistered [ 833.236520] Key type ._llcrypt registered [ 833.238815] Key type .llcrypt registered [ 833.552547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 833.585430] alg: No test for adler32 (adler32-zlib) [ 834.505890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.568627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.590823] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 835.473392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.514139] LNet: 47156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.524572] LNetError: 47156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.548303] LNet: Removed LNI 192.168.206.10@tcp [ 835.578704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.622427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 835.672150] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 842.486851] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:25:42 (1787847942) [ 842.633433] LNet: 47738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.645417] LNetError: 47738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.661524] LNet: Removed LNI 192.168.206.10@tcp [ 843.292572] Key type .llcrypt unregistered [ 843.295017] Key type ._llcrypt unregistered [ 849.883268] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 850.032345] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 850.038757] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 850.050757] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 852.775434] Key type ._llcrypt registered [ 852.777663] Key type .llcrypt registered [ 852.939404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 852.967994] alg: No test for adler32 (adler32-zlib) [ 853.908906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.978748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 854.008299] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 854.958332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 854.984500] LNet: 48203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.995513] LNetError: 48203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.032760] LNet: Removed LNI 192.168.206.10@tcp [ 855.056277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.118669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 855.139794] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 861.099589] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:26:01 (1787847961) [ 861.264454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.293549] LNet: 48801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.302539] LNetError: 48801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.323166] LNet: Removed LNI 192.168.206.10@tcp [ 861.352214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.431951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 867.108806] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:26:06 (1787847966) [ 867.166233] LNet: 49376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.800331] Key type .llcrypt unregistered [ 867.809670] Key type ._llcrypt unregistered [ 874.584789] Key type ._llcrypt registered [ 874.591183] Key type .llcrypt registered [ 875.018249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 875.046779] alg: No test for adler32 (adler32-zlib) [ 876.101457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.159390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 876.176773] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 877.328301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.356270] LNet: 49784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.362782] LNetError: 49784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.388369] LNet: Removed LNI 192.168.206.10@tcp [ 877.443354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.564569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 877.602067] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 884.859698] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:26:24 (1787847984) [ 884.986559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.018549] LNet: 50373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.027578] LNetError: 50373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.057831] LNet: Removed LNI 192.168.206.10@tcp [ 885.128742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.254510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 892.205280] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:26:31 (1787847991) [ 893.771612] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 895.628609] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:26:35 (1787847995) [ 896.738801] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 898.608989] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:26:38 (1787847998) [ 900.246325] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 902.022690] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:26:41 (1787848001) [ 903.905691] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 905.460134] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:26:45 (1787848005) [ 906.829990] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 908.204187] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:26:48 (1787848008) [ 908.325173] LNet: 52428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.009681] Key type .llcrypt unregistered [ 909.019294] Key type ._llcrypt unregistered [ 914.489856] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 914.605299] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 914.607284] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 914.613465] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 917.223497] Key type ._llcrypt registered [ 917.225261] Key type .llcrypt registered [ 917.496558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 917.522570] alg: No test for adler32 (adler32-zlib) [ 918.457482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.565546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 918.604137] LNet: Added LNI 192.168.206.10@tcp [8/512/0/180] [ 918.611233] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 918.613087] LNet: 52756:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 918.740354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.758427] LNet: 52775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.762166] LNetError: 52775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 918.768848] LNet: Removed LNI 10.1.2.3@tcp [ 918.771333] LNetError: 52775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.789311] LNet: Removed LNI 192.168.206.10@tcp [ 918.821867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.885274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 918.912386] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 918.994327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.023313] LNet: 52799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.029253] LNetError: 52799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.040137] LNet: Removed LNI 192.168.206.10@tcp [ 919.078769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.181574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 919.215732] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 919.220237] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.228106] LNet: 52807:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 919.393541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.422296] LNet: 52825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.442945] LNetError: 52825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 919.455195] LNet: Removed LNI 10.1.2.3@tcp [ 919.456615] LNetError: 52825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.494338] LNet: Removed LNI 192.168.206.10@tcp [ 919.543405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.649975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 919.739187] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 919.753362] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.756354] LNet: 52834:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 920.010517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.039524] LNet: 52860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.051629] LNetError: 52860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 920.062723] LNet: Removed LNI 10.1.2.3@tcp [ 920.071200] LNetError: 52860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.101200] LNet: Removed LNI 192.168.206.10@tcp [ 920.137165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.233289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 920.261716] LNet: Added LNI 192.168.206.10@tcp [8/256/0/182] [ 920.272684] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 920.402654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.422535] LNet: 52888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.433066] LNetError: 52888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 920.450328] LNet: Removed LNI 10.1.2.3@tcp [ 920.452336] LNetError: 52888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.470850] LNet: Removed LNI 192.168.206.10@tcp [ 920.527445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.633642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 920.648181] LNet: Added LNI 192.168.206.10@tcp [8/256/0/182] [ 920.656524] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 920.664654] LNet: 52896:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 920.770634] LNet: Removed LNI 10.1.2.3@tcp [ 920.786751] LNet: Removed LNI 192.168.206.10@tcp [ 921.555290] Key type .llcrypt unregistered [ 921.559274] Key type ._llcrypt unregistered [ 934.192334] Key type ._llcrypt registered [ 934.198215] Key type .llcrypt registered [ 935.408149] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 12:27:15 (1787848035) [ 936.752807] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 938.543804] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:27:18 (1787848038) [ 938.875848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 938.884587] alg: No test for adler32 (adler32-zlib) [ 939.871849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.922887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 940.000919] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 940.714512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.10 [ 940.728754] LNet: 54179:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 940.735912] LNet: Removed LNI 192.168.206.10@tcp [ 946.577919] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:27:26 (1787848046) [ 946.680749] LNet: 54741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.696486] LNetError: 54741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.725941] LNet: Removed LNI 192.168.206.10@tcp [ 947.336555] Key type .llcrypt unregistered [ 947.341616] Key type ._llcrypt unregistered [ 952.616612] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 952.775170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 955.398848] Key type ._llcrypt registered [ 955.400384] Key type .llcrypt registered [ 955.627439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 955.644644] alg: No test for adler32 (adler32-zlib) [ 957.140339] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 964.493283] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:27:43 (1787848063) [ 964.613917] LNet: 55659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.628187] LNetError: 55659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 964.700725] LNet: Removed LNI 10.1.2.3@tcp [ 965.586609] Key type .llcrypt unregistered [ 965.591394] Key type ._llcrypt unregistered [ 974.438224] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 974.647721] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 974.654387] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 974.669421] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 977.483902] Key type ._llcrypt registered [ 977.486208] Key type .llcrypt registered [ 977.728854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 977.773238] alg: No test for adler32 (adler32-zlib) [ 978.766738] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 985.045691] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:28:05 (1787848085) [ 986.645838] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 988.444473] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:28:08 (1787848088) [ 988.622096] LNet: 56852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.630954] LNetError: 56852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 988.684570] LNet: Removed LNI 10.1.2.3@tcp [ 989.524160] Key type .llcrypt unregistered [ 989.531613] Key type ._llcrypt unregistered [ 997.157157] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 997.480809] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 997.489754] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 997.499323] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1000.387458] Key type ._llcrypt registered [ 1000.389459] Key type .llcrypt registered [ 1000.598925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1000.617265] alg: No test for adler32 (adler32-zlib) [ 1001.538504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1001.631055] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1010.413648] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:28:30 (1787848110) [ 1010.943215] LNet: 58092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.950876] LNetError: 58092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1010.969138] LNet: Removed LNI 10.1.2.3@tcp [ 1011.725738] Key type .llcrypt unregistered [ 1011.731324] Key type ._llcrypt unregistered [ 1018.330225] Key type ._llcrypt registered [ 1018.336594] Key type .llcrypt registered [ 1018.596534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1018.619629] alg: No test for adler32 (adler32-zlib) [ 1019.540994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1019.561753] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1027.041446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1027.706263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_interrupt -r 1 [ 1027.782535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_interrupt -r 1 [ 1027.851418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1027.879631] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1028.105863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1028.139788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1028.178396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1028.228634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1028.268817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1028.302088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1028.353331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1028.399416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1028.420305] LNet: 58533:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1028.466615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1028.511937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1028.533480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1028.558537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1028.599117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1028.652647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1028.687640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1029.214681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_dropped -r 1 [ 1029.249665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_dropped -r 1 [ 1029.282788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1029.304715] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1029.561695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1029.586981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1029.618077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1029.657174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1029.702689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1029.740130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1029.792580] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1029.794166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1029.830730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1029.849918] LNet: 58643:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1029.875533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1029.913079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1029.940373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1029.993218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1030.036655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1030.089234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1030.123988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1030.672967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_aborted -r 1 [ 1030.715490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_aborted -r 1 [ 1030.756536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1030.791767] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1031.076644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1031.127742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1031.160532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1031.189810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1031.227305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1031.273137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1031.312925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1031.359344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1031.377297] LNet: 58752:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1031.395780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1031.448092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1031.495211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1031.539860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1031.598574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1031.648688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1031.703405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1032.380101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_no_route -r 1 [ 1032.461935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_no_route -r 1 [ 1032.526794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1032.787224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1032.820564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1032.841141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1032.873445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1032.898436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1032.927684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1032.962333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.998132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1033.031949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1033.071534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1033.102553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1033.129534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1033.161393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1033.186811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1033.219066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1033.639419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_timeout -r 1 [ 1033.671709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_timeout -r 1 [ 1033.700311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1033.721312] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1033.732191] LNet: Skipped 2 previous similar messages [ 1034.023537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1034.081951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1034.113395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1034.135334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1034.161710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1034.201521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1034.254192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.293202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1034.328225] LNet: 58970:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1034.341169] LNet: 58970:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1034.376697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.405248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.443688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1034.473315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1034.500371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.524532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1034.556615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1035.053117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_error -r 1 [ 1035.115594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_error -r 1 [ 1035.188372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1035.532619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1035.592200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1035.632968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1035.678206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1035.740320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1035.771701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1035.807811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1035.845457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1035.869632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1035.900732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1035.927372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1035.950510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1035.972358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1035.994894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1036.015211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1036.225723] LNet: 59124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.229979] LNetError: 59124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.285247] LNet: Removed LNI 192.168.206.10@tcp [ 1037.748487] Key type .llcrypt unregistered [ 1037.750917] Key type ._llcrypt unregistered [ 1055.520763] Key type ._llcrypt registered [ 1055.523430] Key type .llcrypt registered [ 1056.798800] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:29:16 (1787848156) [ 1057.671725] Key type .llcrypt unregistered [ 1057.673705] Key type ._llcrypt unregistered [ 1063.316722] Key type ._llcrypt registered [ 1063.318869] Key type .llcrypt registered [ 1063.572609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1063.595337] alg: No test for adler32 (adler32-zlib) [ 1064.565494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1064.596316] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1074.324441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1088.424319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1088.496281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1088.561960] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1090.268637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_interrupt -r 1 [ 1090.327899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_interrupt -r 1 [ 1090.389213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_interrupt -r 1 [ 1090.449249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_interrupt -r 1 [ 1090.529184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_interrupt -r 1 [ 1090.577417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_interrupt -r 1 [ 1090.600191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_interrupt -r 1 [ 1090.626955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_interrupt -r 1 [ 1090.677464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1090.706919] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1090.713487] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1091.043953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1091.083036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1091.116942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1091.163428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1091.218300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1091.255202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1091.297216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1091.329931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1091.344644] LNet: 60596:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1091.370863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1091.420219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1091.475776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1091.526678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1091.573227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1091.620727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1091.719111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1092.404888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_dropped -r 1 [ 1092.441338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_dropped -r 1 [ 1092.490622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_dropped -r 1 [ 1092.545940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_dropped -r 1 [ 1092.603296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_dropped -r 1 [ 1092.662626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_dropped -r 1 [ 1092.693762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_dropped -r 1 [ 1092.721074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_dropped -r 1 [ 1092.755501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1092.773363] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1092.778713] LNet: Skipped 2 previous similar messages [ 1092.983854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1093.034324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1093.079923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1093.104382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1093.148526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1093.169096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1093.195684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1093.222787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1093.240440] LNet: 60718:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1093.254974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1093.282356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1093.310818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1093.349429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1093.392297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1093.485259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1093.575543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1094.280150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_aborted -r 1 [ 1094.326722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_aborted -r 1 [ 1094.362115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_aborted -r 1 [ 1094.405658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_aborted -r 1 [ 1094.462897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_aborted -r 1 [ 1094.515660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_aborted -r 1 [ 1094.581601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_aborted -r 1 [ 1094.635502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_aborted -r 1 [ 1094.696294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1094.716237] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1094.727408] LNet: Skipped 2 previous similar messages [ 1095.016878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1095.057688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1095.091263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1095.130805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1095.165840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1095.234732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1095.280216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1095.326981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1095.350773] LNet: 60840:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1095.393355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1095.464832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1095.515412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1095.557744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1095.604499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1095.658386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1095.736260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1096.279410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_no_route -r 1 [ 1096.340352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_no_route -r 1 [ 1096.382182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_no_route -r 1 [ 1096.452635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_no_route -r 1 [ 1096.510719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_no_route -r 1 [ 1096.580192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_no_route -r 1 [ 1096.638642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_no_route -r 1 [ 1096.681863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_no_route -r 1 [ 1096.725275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1096.746788] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1096.753206] LNet: Skipped 2 previous similar messages [ 1096.973766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1097.020524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1097.051985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1097.085067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1097.111487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1097.142221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1097.196579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1097.245112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1097.288575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1097.348380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1097.406418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1097.444398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1097.497211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.526458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.557853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1098.200652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_timeout -r 1 [ 1098.236196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_timeout -r 1 [ 1098.263700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_timeout -r 1 [ 1098.288164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_timeout -r 1 [ 1098.315179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_timeout -r 1 [ 1098.338156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_timeout -r 1 [ 1098.385423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_timeout -r 1 [ 1098.434784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_timeout -r 1 [ 1098.476262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1098.697824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1098.745451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1098.794748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1098.834456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1098.883816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1098.933874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1098.975786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1099.013186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1099.032335] LNet: 61082:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1099.037699] LNet: 61082:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1099.057287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1099.098271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1099.139495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1099.177920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1099.222397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1099.274731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1099.318183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1099.885943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_error -r 1 [ 1099.927521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_error -r 1 [ 1099.965212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_error -r 1 [ 1100.023847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_error -r 1 [ 1100.078704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_error -r 1 [ 1100.111767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_error -r 1 [ 1100.141183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_error -r 1 [ 1100.189870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_error -r 1 [ 1100.229251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1100.501274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1100.542252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1100.579802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1100.614498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.640891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.660843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.681168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.708943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.741731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.778807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.829957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.878614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1100.931391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1100.975027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.024374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1106.943488] LNet: 61283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.953536] LNetError: 61283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.971226] LNet: Removed LNI 192.168.206.10@tcp [ 1106.994965] LNet: Removed LNI 192.168.206.10@tcp1 [ 1107.380566] Key type .llcrypt unregistered [ 1107.382169] Key type ._llcrypt unregistered [ 1124.974816] Key type ._llcrypt registered [ 1124.980961] Key type .llcrypt registered [ 1126.437898] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:30:25 (1787848225) [ 1127.333294] Key type .llcrypt unregistered [ 1127.338620] Key type ._llcrypt unregistered [ 1134.126673] Key type ._llcrypt registered [ 1134.129786] Key type .llcrypt registered [ 1134.304341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1134.336134] alg: No test for adler32 (adler32-zlib) [ 1135.233557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1135.262396] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1143.643733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1144.447449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e remote_dropped -r 1 [ 1144.483945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e remote_dropped -r 1 [ 1144.521993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1144.541667] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1144.789470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1144.821557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1144.855558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1144.881168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1144.903746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1144.951677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1145.000310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1145.047537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1145.061449] LNet: 62537:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1145.096398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1145.127398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1145.150866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1145.177940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1145.217828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1145.257794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1145.306930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1145.828979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e remote_error -r 1 [ 1145.875505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e remote_error -r 1 [ 1145.911521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1145.927824] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1146.208688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1146.248697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1146.285355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1146.330573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1146.402716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1146.447085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1146.497527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1146.548942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1146.578349] LNet: 62646:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1146.612594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1146.651716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1146.702624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1146.739082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1146.779727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1146.813417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1146.859615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1147.499894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e remote_timeout -r 1 [ 1147.569482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e remote_timeout -r 1 [ 1147.643749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1147.693593] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1147.912774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1147.961709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1147.999316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1148.026986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1148.058748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1148.093899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1148.124572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1148.165743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1148.206776] LNet: 62755:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1148.234064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1148.269462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1148.312889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1148.352591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1148.397979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1148.466912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1148.518876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1148.863755] LNet: 62800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.887139] LNetError: 62800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.923265] LNet: Removed LNI 192.168.206.10@tcp [ 1149.399602] Key type .llcrypt unregistered [ 1149.405457] Key type ._llcrypt unregistered [ 1164.765928] Key type ._llcrypt registered [ 1164.768127] Key type .llcrypt registered [ 1166.045525] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:31:05 (1787848265) [ 1166.802460] Key type .llcrypt unregistered [ 1166.807808] Key type ._llcrypt unregistered [ 1174.051577] Key type ._llcrypt registered [ 1174.054166] Key type .llcrypt registered [ 1174.393938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1174.409741] alg: No test for adler32 (adler32-zlib) [ 1175.330219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1175.351718] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1183.094108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1192.798839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1192.848403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1192.893471] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1194.473373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e remote_dropped -r 1 [ 1194.513953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e remote_dropped -r 1 [ 1194.546886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e remote_dropped -r 1 [ 1194.583357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e remote_dropped -r 1 [ 1194.630643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e remote_dropped -r 1 [ 1194.675351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e remote_dropped -r 1 [ 1194.721816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e remote_dropped -r 1 [ 1194.768659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e remote_dropped -r 1 [ 1194.827368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1194.857733] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1194.864431] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.110@tcp [ 1195.112879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1195.137212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1195.176296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1195.213575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1195.268446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1195.326449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1195.384786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1195.433750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1195.464285] LNet: 64270:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1195.489795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1195.518576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1195.555617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1195.612449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1195.641629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1195.666779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1195.694727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1195.725572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1196.224136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e remote_error -r 1 [ 1196.259428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e remote_error -r 1 [ 1196.302528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e remote_error -r 1 [ 1196.339337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e remote_error -r 1 [ 1196.379723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e remote_error -r 1 [ 1196.413649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e remote_error -r 1 [ 1196.443326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e remote_error -r 1 [ 1196.489601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e remote_error -r 1 [ 1196.530988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1196.555960] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1196.562626] LNet: Skipped 2 previous similar messages [ 1196.789770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1196.831665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1196.863953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1196.918533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1196.963214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.993727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1197.026635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1197.065439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1197.092578] LNet: 64394:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1197.119537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1197.166655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1197.216589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1197.256500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1197.304755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.348134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1197.377907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1197.417578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.977381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e remote_timeout -r 1 [ 1198.008857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e remote_timeout -r 1 [ 1198.047541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e remote_timeout -r 1 [ 1198.077913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e remote_timeout -r 1 [ 1198.119426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e remote_timeout -r 1 [ 1198.153041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e remote_timeout -r 1 [ 1198.189608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e remote_timeout -r 1 [ 1198.231601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e remote_timeout -r 1 [ 1198.286151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1198.309982] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1198.573093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1198.621390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1198.679468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1198.723820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1198.774542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1198.811595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1198.844565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1198.887142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1198.916499] LNet: 64517:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1198.939485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1198.973413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.021986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1199.057485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1199.108065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.161364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1199.205622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1199.247426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1204.285678] LNet: 64599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1204.311439] LNetError: 64599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1204.328929] LNet: Removed LNI 192.168.206.10@tcp [ 1204.348163] LNet: Removed LNI 192.168.206.10@tcp1 [ 1204.779801] Key type .llcrypt unregistered [ 1204.782907] Key type ._llcrypt unregistered [ 1220.971318] Key type ._llcrypt registered [ 1220.974170] Key type .llcrypt registered [ 1222.523693] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:32:01 (1787848321) [ 1223.835911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1225.539669] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:32:05 (1787848325) [ 1226.227858] Key type .llcrypt unregistered [ 1226.234887] Key type ._llcrypt unregistered [ 1231.996904] Key type ._llcrypt registered [ 1232.001773] Key type .llcrypt registered [ 1232.236705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1232.269534] alg: No test for adler32 (adler32-zlib) [ 1233.168422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1233.208986] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1240.898741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1241.534992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e network_timeout -r 1 [ 1241.559986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e network_timeout -r 1 [ 1241.586472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1241.600838] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1241.789954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1241.820238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1241.847364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.880508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.907439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.927855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.951777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.974968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.989808] LNet: 66146:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1242.002425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1242.026675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1242.048419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1242.080869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1242.117893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1242.142221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1242.427649] LNet: 66189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1242.432467] LNetError: 66189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1242.448677] LNet: Removed LNI 192.168.206.10@tcp [ 1242.806530] Key type .llcrypt unregistered [ 1242.811337] Key type ._llcrypt unregistered [ 1261.968535] Key type ._llcrypt registered [ 1261.974231] Key type .llcrypt registered [ 1262.257778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1262.273989] alg: No test for adler32 (adler32-zlib) [ 1263.210186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1263.235600] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1270.449130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1278.866489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.902359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1278.917706] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1280.136879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e network_timeout -r 1 [ 1280.180108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e network_timeout -r 1 [ 1280.219217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e network_timeout -r 1 [ 1280.252282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e network_timeout -r 1 [ 1280.289861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e network_timeout -r 1 [ 1280.334344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e network_timeout -r 1 [ 1280.368139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e network_timeout -r 1 [ 1280.398529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e network_timeout -r 1 [ 1280.431904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1280.446931] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1280.662771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1280.672243] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1280.683419] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.110@tcp [ 1280.695596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1280.740666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1280.785709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.819625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.870775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.907691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.973963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1281.004297] LNet: 67072:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1281.042297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1281.125140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1281.193876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1281.258826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1281.301558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1281.339505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1286.268961] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1286.653889] LNet: 67150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1286.663040] LNetError: 67150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1286.672649] LNet: Removed LNI 192.168.206.10@tcp [ 1286.684172] LNet: Removed LNI 192.168.206.10@tcp1 [ 1287.101441] Key type .llcrypt unregistered [ 1287.104113] Key type ._llcrypt unregistered [ 1303.597346] Key type ._llcrypt registered [ 1303.600342] Key type .llcrypt registered [ 1304.873214] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:33:24 (1787848404) [ 1305.628910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1305.634627] alg: No test for adler32 (adler32-zlib) [ 1306.521927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1306.581514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1306.619969] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1307.234760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1307.248051] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1307.685667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 1307.717885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1307.743698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1307.763860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1307.782558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1307.799848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1307.816534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.10@tcp [ 1307.828216] LNetError: 68249:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.10@tcp to recovery [ 1308.639658] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1308.640056] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1310.687359] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1311.320910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1314.783573] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1315.918702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1315.943988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1315.989582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1316.005929] LNet: 68367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1316.011757] LNetError: 68367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1317.023567] LNet: Removed LNI 192.168.206.10@tcp [ 1317.028925] LNetError: 68367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1317.045080] LNet: Removed LNI 192.168.206.10@tcp1 [ 1317.068783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1317.109236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1317.130583] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1317.739986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1317.759439] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1318.397973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 1318.466496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1318.532832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1318.575819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1318.610594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1318.641896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1318.675118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1318.712674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.10@tcp [ 1318.731558] LNetError: 68639:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.10@tcp to recovery [ 1319.135392] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1322.268864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1325.282389] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1325.293131] LNet: Skipped 1 previous similar message [ 1330.158793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1330.195799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1330.230068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1336.480833] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:33:56 (1787848436) [ 1336.596595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1336.640766] LNet: 69351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.647840] LNetError: 69351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.657769] LNet: Removed LNI 192.168.206.10@tcp [ 1336.675336] LNet: Removed LNI 192.168.206.10@tcp1 [ 1336.721371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1336.774612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1336.799287] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1337.369354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1337.395787] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1338.201284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 1338.274475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1338.316991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1338.354941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1338.395829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1338.452433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1338.502240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.10@tcp --health 0 [ 1338.627080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1338.783140] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.10@tcp [ 1339.808754] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1339.820207] LNet: Skipped 1 previous similar message [ 1354.259797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1354.422522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1354.454447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1354.477507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.10@tcp --health 500 [ 1354.550776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1361.231383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1361.301803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1361.333776] LNet: 69849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.343776] LNetError: 69849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1361.353090] LNetError: 69849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1362.399309] LNet: Removed LNI 192.168.206.10@tcp [ 1362.414397] LNet: Removed LNI 192.168.206.10@tcp1 [ 1362.466510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.512646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1362.538449] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1363.189309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1363.210774] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1364.015711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 1364.054743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1364.105358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1364.129577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1364.157438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1364.190198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1364.215802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1364.238234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.10@tcp --health 0 [ 1365.600160] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1365.613590] LNet: Skipped 3 previous similar messages [ 1366.690644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1376.747767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1376.765698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1376.782567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1383.143971] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:34:42 (1787848482) [ 1383.531420] LNet: 70906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.548388] LNetError: 70906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.552703] LNetError: 70906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1383.557721] LNet: Removed LNI 192.168.206.10@tcp [ 1383.571869] LNet: Removed LNI 192.168.206.10@tcp1 [ 1384.143804] Key type .llcrypt unregistered [ 1384.147177] Key type ._llcrypt unregistered [ 1390.472168] Key type ._llcrypt registered [ 1390.474820] Key type .llcrypt registered [ 1390.779292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1390.800327] alg: No test for adler32 (adler32-zlib) [ 1391.894721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1391.953353] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1401.099115] Lustre: DEBUG MARKER: Initial discovery [ 1401.391653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.110@tcp [ 1403.808201] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1403.998025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1404.054763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.110@tcp [ 1404.080850] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1404.116099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1404.174504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.110@tcp -r 1 -m GET -l 3 [ 1404.220532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.110@tcp -r 1 -m GET -e local_error [ 1404.274415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.110@tcp -r 1 -m PUT -l 6 [ 1404.320525] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1405.588996] Lustre: DEBUG MARKER: Force oleg610-client.virtnet to discover 192.168.206.110@tcp (in background) [ 1405.757530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.110@tcp [ 1406.966925] Lustre: DEBUG MARKER: Force oleg610-server to discover 192.168.206.10@tcp [ 1408.806962] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1408.967381] Lustre: DEBUG MARKER: Wait for 71674 [ 1410.350719] Lustre: DEBUG MARKER: Finished wait on 71674 [ 1410.514017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1410.542404] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1410.558950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1410.800783] LNet: 72126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.817058] LNetError: 72126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.843035] LNet: Removed LNI 192.168.206.10@tcp [ 1411.233474] Key type .llcrypt unregistered [ 1411.242539] Key type ._llcrypt unregistered [ 1426.475879] Key type ._llcrypt registered [ 1426.480816] Key type .llcrypt registered [ 1427.740433] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:35:27 (1787848527) [ 1428.247419] Key type .llcrypt unregistered [ 1428.252316] Key type ._llcrypt unregistered [ 1434.703023] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1434.926532] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1434.936692] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1434.943255] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1437.470326] Key type ._llcrypt registered [ 1437.472824] Key type .llcrypt registered [ 1437.703897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1437.727723] alg: No test for adler32 (adler32-zlib) [ 1438.719544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1438.795176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1438.827527] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1439.625991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1439.659259] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1447.334596] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:35:47 (1787848547) [ 1447.486286] LNet: 74217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.495308] LNetError: 74217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1447.503826] LNet: Removed LNI 10.1.2.3@tcp [ 1447.516234] LNet: Removed LNI 192.168.206.10@tcp [ 1448.209500] Key type .llcrypt unregistered [ 1448.220574] Key type ._llcrypt unregistered [ 1455.044986] Key type ._llcrypt registered [ 1455.050786] Key type .llcrypt registered [ 1455.290402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1455.312695] alg: No test for adler32 (adler32-zlib) [ 1456.254372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1456.309308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1456.326408] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1456.909687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1456.940804] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1457.697971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.10@tcp [ 1458.722485] LNetError: Unexpected error -22 connecting to 192.168.206.10@tcp at host 192.168.206.10:988 [ 1463.278736] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:36:03 (1787848563) [ 1463.445966] LNet: 75357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.455252] LNetError: 75357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1463.467359] LNet: Removed LNI 10.1.2.3@tcp [ 1463.485686] LNet: Removed LNI 192.168.206.10@tcp [ 1464.097383] Key type .llcrypt unregistered [ 1464.102283] Key type ._llcrypt unregistered [ 1470.010455] Key type ._llcrypt registered [ 1470.022101] Key type .llcrypt registered [ 1470.344161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1470.360990] alg: No test for adler32 (adler32-zlib) [ 1471.271117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.320512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1471.353396] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1472.081471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1472.105640] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 1472.919783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.10@tcp1 --nid 192.168.206.10@tcp2 [ 1477.756900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp1 192.168.206.10@tcp2 [ 1477.775614] LNetError: 76199:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.10@tcp1 to 192.168.206.10@tcp2 - Route table may be misconfigured [ 1477.798245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp2 192.168.206.10@tcp1 [ 1483.011643] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:36:23 (1787848583) [ 1483.163717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1483.184726] LNet: 76772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1483.194354] LNetError: 76772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1483.196687] LNet: Removed LNI 192.168.206.10@tcp1 [ 1484.264799] LNet: Removed LNI 192.168.206.10@tcp2 [ 1484.314796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1484.394983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1484.442970] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1485.141094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1485.157090] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1485.968085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 1486.031756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp 192.168.206.10@tcp [ 1486.080951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp1 192.168.206.10@tcp1 [ 1486.138142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.10@tcp -d 192.168.206.10@tcp -e network_timeout [ 1486.180622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -e network_timeout [ 1486.213256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -e network_timeout [ 1486.260600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -e network_timeout [ 1486.309711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1486.342394] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1486.363481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1486.431085] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1486.480496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1486.574525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1492.434284] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:36:32 (1787848592) [ 1492.678079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.697595] LNet: 77633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.708970] LNetError: 77633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.714677] LNetError: 77633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1492.720933] LNet: Removed LNI 192.168.206.10@tcp [ 1493.737808] LNet: Removed LNI 192.168.206.10@tcp1 [ 1493.761742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1493.867088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1494.184811] LNet: 77674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.628457] Key type .llcrypt unregistered [ 1494.629986] Key type ._llcrypt unregistered [ 1505.510330] Key type ._llcrypt registered [ 1505.512381] Key type .llcrypt registered [ 1506.671585] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:36:46 (1787848606) [ 1507.131987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1507.143525] alg: No test for adler32 (adler32-zlib) [ 1508.088271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.524839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1511.102567] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:36:51 (1787848611) [ 1511.331399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.365751] LNet: 78772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.393167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.500248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1511.548951] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1512.185203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1512.212034] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1513.075842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1513.104961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp1 [ 1513.131551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp1 [ 1519.716217] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:36:59 (1787848619) [ 1521.630928] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg610-server [ 1523.382286] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:37:03 (1787848623) [ 1525.248830] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg610-server [ 1526.871798] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:37:06 (1787848626) [ 1528.850864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg610-server [ 1530.383538] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:37:10 (1787848630) [ 1532.185603] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg610-server [ 1533.666589] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:37:13 (1787848633) [ 1535.174948] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg610-server [ 1536.864793] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:37:16 (1787848636) [ 1538.635154] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg610-server [ 1540.093809] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:37:20 (1787848640) [ 1541.675700] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg610-server [ 1543.185562] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:37:23 (1787848643) [ 1544.615754] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg610-server [ 1546.192689] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:37:26 (1787848646) [ 1548.011140] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg610-server [ 1549.779347] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:37:29 (1787848649) [ 1549.917097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1549.958445] LNet: 82648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.966601] LNetError: 82648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.984884] LNet: Removed LNI 192.168.206.10@tcp [ 1551.015169] LNet: Removed LNI 192.168.206.10@tcp1 [ 1551.038892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.106882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1551.131702] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1551.988793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1552.053344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1552.236181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.268497] LNet: 82797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.275334] LNetError: 82797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.287508] LNetError: 82797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1552.310680] LNet: Removed LNI 192.168.206.10@tcp [ 1552.342140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.395888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1552.415371] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1553.122956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1553.392389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1553.416357] LNetError: 82942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.469659] LNet: Removed LNI 192.168.206.10@tcp [ 1554.490827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.564933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1554.618509] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1555.288129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1555.359530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1555.541733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1555.576377] LNet: 83089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.581456] LNet: 83089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1555.589904] LNetError: 83089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.606224] LNet: Removed LNI 192.168.206.10@tcp [ 1555.660928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.737662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1555.771705] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1556.431378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1556.580546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.611094] LNet: Removed LNI 192.168.206.10@tcp [ 1557.655520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.723685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1557.751345] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1558.401155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1558.500679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1558.739371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.783508] LNet: Removed LNI 192.168.206.10@tcp [ 1559.797343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.843688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1559.867519] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1560.718714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1560.918315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.940730] LNet: 83527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.950573] LNet: 83527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1560.956329] LNetError: 83527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1560.969092] LNetError: 83527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1560.989307] LNet: Removed LNI 192.168.206.10@tcp [ 1561.014916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.055244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1561.093665] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1561.886239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1561.995560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1562.244867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.305643] LNet: Removed LNI 192.168.206.10@tcp [ 1563.330318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.382878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1563.404864] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1564.048074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1564.313843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.349065] LNet: Removed LNI 192.168.206.10@tcp [ 1565.390359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.467464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1565.488554] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1566.086264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1566.201669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1566.422697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.465976] LNet: Removed LNI 192.168.206.10@tcp [ 1567.488067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.547873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1567.572881] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1568.442511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1569.922000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.962260] LNet: 84122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.969488] LNet: 84122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1569.978668] LNetError: 84122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.982786] LNetError: 84122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1571.049565] LNet: Removed LNI 192.168.206.10@tcp [ 1571.107226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.189163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1571.240606] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1572.068341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1572.128140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1572.395374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.490517] LNet: Removed LNI 192.168.206.10@tcp [ 1573.540475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.625815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1573.658445] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1574.481272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1576.006874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.069558] LNet: Removed LNI 192.168.206.10@tcp [ 1577.100379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.173691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1577.226246] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1578.011802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1578.078743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.109377] LNet: Removed LNI 192.168.206.10@tcp [ 1578.125596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.168618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.185493] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1578.924885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1578.998775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.033529] LNet: Removed LNI 192.168.206.10@tcp [ 1579.057543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.119390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1579.203431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.237639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.310055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1579.436933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.471103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.559183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1579.633129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.654560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.698215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1579.716666] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1580.391610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1580.423918] LNet: Removed LNI 192.168.206.10@tcp [ 1580.437927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.515382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1587.050528] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:38:06 (1787848686) [ 1587.182271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.217151] LNet: 85441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.228959] LNet: 85441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1587.284476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.362092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1587.401079] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/180] [ 1588.507333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.551514] LNetError: 85583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.556025] LNetError: 85583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1588.576979] LNet: Removed LNI 192.168.206.10@tcp231 [ 1588.614347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.688598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1588.717092] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/180] [ 1588.722291] LNet: 85591:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1589.110533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1589.141604] LNet: Removed LNI 192.168.206.10@tcp231 [ 1589.154135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1589.185850] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/180] [ 1590.275860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.311812] LNet: Removed LNI 192.168.206.10@tcp231 [ 1590.367349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.457398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1590.485276] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/0] [ 1590.490533] LNet: 85744:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1590.975693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1591.007439] LNet: Removed LNI 192.168.206.10@tcp231 [ 1591.028450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1591.049574] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/0] [ 1592.233659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.279315] LNet: Removed LNI 192.168.206.10@tcp231 [ 1592.297435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.368621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1592.414347] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/60] [ 1592.420424] LNet: 85895:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1592.823947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1592.856543] LNet: Removed LNI 192.168.206.10@tcp231 [ 1592.873897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1592.903342] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/60] [ 1594.030026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.060893] LNet: Removed LNI 192.168.206.10@tcp231 [ 1594.084575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.165581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1594.215853] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/180] [ 1594.680390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1594.724093] LNet: Removed LNI 192.168.206.10@tcp231 [ 1594.744959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1594.778897] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/180] [ 1601.674680] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:38:21 (1787848701) [ 1601.919547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.958644] LNet: Removed LNI 192.168.206.10@tcp231 [ 1601.979344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1602.054663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1602.081629] LNet: Added LNI 192.168.206.10@tcp232 [8/256/0/180] [ 1602.951717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1602.992889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1603.022692] LNet: Removed LNI 192.168.206.10@tcp232 [ 1603.045665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1603.072840] LNet: Added LNI 192.168.206.10@tcp232 [8/256/0/180] [ 1603.985458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1604.060224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1604.101937] LNet: Removed LNI 192.168.206.10@tcp232 [ 1604.141354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1604.192809] LNet: Added LNI 192.168.206.10@tcp232 [8/256/0/180] [ 1604.970870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1605.017760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1605.064291] LNet: Removed LNI 192.168.206.10@tcp232 [ 1605.087732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1605.110158] LNet: Added LNI 192.168.206.10@tcp232 [8/256/0/180] [ 1605.847194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1605.894768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1605.939745] LNet: Removed LNI 192.168.206.10@tcp232 [ 1605.957653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1612.267845] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:38:32 (1787848712) [ 1613.485431] Key type .llcrypt unregistered [ 1613.491342] Key type ._llcrypt unregistered [ 1619.517184] Key type ._llcrypt registered [ 1619.527524] Key type .llcrypt registered [ 1619.746567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1619.768736] alg: No test for adler32 (adler32-zlib) [ 1620.786987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1620.821546] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1628.251770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1638.194748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.240807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1638.280674] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1640.036405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_interrupt -r 2 [ 1640.070929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_interrupt -r 2 [ 1640.118594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_interrupt -r 2 [ 1640.165712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_interrupt -r 2 [ 1640.191480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_interrupt -r 2 [ 1640.237239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_interrupt -r 2 [ 1640.274387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_interrupt -r 2 [ 1640.328561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_interrupt -r 2 [ 1640.374020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp 192.168.206.110@tcp [ 1640.395218] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1640.409619] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1640.695989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1640.736390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1640.766616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1640.799144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1640.876597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1640.943548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1640.983601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1641.025350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1641.055335] LNet: 88551:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1641.075611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1641.136959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1641.214948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1641.292745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1641.351077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1641.404687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1641.974902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_dropped -r 2 [ 1642.021287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_dropped -r 2 [ 1642.085467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_dropped -r 2 [ 1642.144395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_dropped -r 2 [ 1642.216044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_dropped -r 2 [ 1642.282750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_dropped -r 2 [ 1642.316485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_dropped -r 2 [ 1642.386513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_dropped -r 2 [ 1642.469406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp 192.168.206.110@tcp [ 1642.504359] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1642.828233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1642.853944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1642.887193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1642.924680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1642.969451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1643.014076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1643.044337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1643.086865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1643.119396] LNet: 88672:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1643.141951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1643.168548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1643.208986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1643.241387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1643.267958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1643.305755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1643.879065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_aborted -r 2 [ 1643.927306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_aborted -r 2 [ 1643.980449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_aborted -r 2 [ 1644.043941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_aborted -r 2 [ 1644.076888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_aborted -r 2 [ 1644.118189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_aborted -r 2 [ 1644.175990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_aborted -r 2 [ 1644.240866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_aborted -r 2 [ 1644.288402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp 192.168.206.110@tcp [ 1644.312835] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1644.596666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1644.640581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1644.683927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1644.705684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1644.733798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1644.780173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1644.825204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1644.857718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1644.894030] LNet: 88792:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1644.927453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1644.960147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1644.990743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1645.031211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1645.063816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1645.092737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1645.629775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_no_route -r 2 [ 1645.677264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_no_route -r 2 [ 1645.724677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_no_route -r 2 [ 1645.773994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_no_route -r 2 [ 1645.815195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_no_route -r 2 [ 1645.857956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_no_route -r 2 [ 1645.893742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_no_route -r 2 [ 1645.954244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_no_route -r 2 [ 1646.001691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp 192.168.206.110@tcp [ 1646.291654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1646.320920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1646.349265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1646.375643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1646.402555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1646.429196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1646.460546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1646.484092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1646.516066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1646.551524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1646.623516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1646.681843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1646.788945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1646.827163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1647.335368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_timeout -r 2 [ 1647.375957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_timeout -r 2 [ 1647.411071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_timeout -r 2 [ 1647.447231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_timeout -r 2 [ 1647.516789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_timeout -r 2 [ 1647.600587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_timeout -r 2 [ 1647.658692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_timeout -r 2 [ 1647.694738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_timeout -r 2 [ 1647.752485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp 192.168.206.110@tcp [ 1647.788028] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1647.794993] LNet: Skipped 1 previous similar message [ 1648.049198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1648.087780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1648.133521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1648.186694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1648.262376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1648.318357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1648.344573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1648.383764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1648.418255] LNet: 89033:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1648.431208] LNet: 89033:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1648.450337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1648.475663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1648.517593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1648.552158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1648.593481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1648.633885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1649.256463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e remote_dropped -r 2 [ 1649.300754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e remote_dropped -r 2 [ 1649.350909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e remote_dropped -r 2 [ 1649.402493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e remote_dropped -r 2 [ 1649.466251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e remote_dropped -r 2 [ 1649.505431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e remote_dropped -r 2 [ 1649.544677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e remote_dropped -r 2 [ 1649.582684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e remote_dropped -r 2 [ 1649.630573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp 192.168.206.110@tcp [ 1649.657379] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.110@tcp [ 1649.894852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1649.917581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1649.945402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1649.973379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1650.010112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1650.057501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1650.098927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1650.136640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1650.174796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1650.207309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1650.236949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1650.272924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1650.313569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1650.357550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1656.211410] LNet: 89231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.221167] LNetError: 89231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1656.230311] LNet: Removed LNI 192.168.206.10@tcp [ 1656.261799] LNet: Removed LNI 192.168.206.10@tcp1 [ 1656.749641] Key type .llcrypt unregistered [ 1656.756038] Key type ._llcrypt unregistered [ 1672.655295] Key type ._llcrypt registered [ 1672.657518] Key type .llcrypt registered [ 1673.704806] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:39:33 (1787848773) [ 1674.222530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1674.234856] alg: No test for adler32 (adler32-zlib) [ 1675.278292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.365776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1675.401099] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1676.267994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1676.317214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.10@tcp [ 1676.337745] LNetError: 90211:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.10@tcp to recovery [ 1676.464842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.10@tcp [ 1676.620528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1682.777504] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:39:42 (1787848782) [ 1683.164650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.211902] LNet: 90802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.231439] LNetError: 90802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.256311] LNet: Removed LNI 192.168.206.10@tcp [ 1683.303955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.379963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1683.423993] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1683.909845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1683.925552] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1684.766799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.10@tcp [ 1684.791956] LNetError: 91068:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.10@tcp to recovery [ 1685.409631] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1685.412659] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1686.005371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.10@tcp [ 1694.812970] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:39:54 (1787848794) [ 1696.163598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg610-server [ 1697.643292] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:39:57 (1787848797) [ 1699.357438] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg610-server [ 1700.911853] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:40:00 (1787848800) [ 1702.603818] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg610-server [ 1704.525959] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:40:04 (1787848804) [ 1704.606304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.644409] LNet: 92692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.653221] LNetError: 92692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.695140] LNet: Removed LNI 192.168.206.10@tcp [ 1705.696624] LNetError: 92692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.719735] LNet: Removed LNI 192.168.206.10@tcp1 [ 1705.765227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.870875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1705.903854] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1706.762790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1706.865927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1706.952920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1707.083461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1707.126478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1707.163378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1713.193991] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:40:13 (1787848813) [ 1713.338067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.368948] LNet: 93418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.386370] LNetError: 93418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.409637] LNet: Removed LNI 192.168.206.10@tcp [ 1713.444936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.515452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1713.545303] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1720.438423] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:40:20 (1787848820) [ 1720.673601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.696715] LNet: 94127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.701830] LNetError: 94127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.711456] LNet: Removed LNI 192.168.206.10@tcp [ 1720.736079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.783418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1720.816430] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1727.791350] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:40:27 (1787848827) [ 1729.318448] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1730.943986] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:40:30 (1787848830) [ 1731.458637] LNet: 95162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.472481] LNetError: 95162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.494981] LNet: Removed LNI 192.168.206.10@tcp [ 1732.133548] Key type .llcrypt unregistered [ 1732.138942] Key type ._llcrypt unregistered [ 1738.401564] Key type ._llcrypt registered [ 1738.403460] Key type .llcrypt registered [ 1738.533680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1738.572559] alg: No test for adler32 (adler32-zlib) [ 1739.455393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1739.493514] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1745.858860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1751.008045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.110@tcp [ 1751.569853] LNet: 95588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1751.577443] LNetError: 95588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.604800] LNet: Removed LNI 192.168.206.10@tcp [ 1752.037110] Key type .llcrypt unregistered [ 1752.049254] Key type ._llcrypt unregistered [ 1764.148897] Key type ._llcrypt registered [ 1764.151093] Key type .llcrypt registered [ 1765.661447] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1767.442884] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1769.280215] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:41:09 (1787848869) [ 1771.012627] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1772.905862] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:41:12 (1787848872) [ 1775.494679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg610-server [ 1777.722164] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:41:17 (1787848877) [ 1780.017504] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg610-server [ 1781.830492] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:41:21 (1787848881) [ 1783.410409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1783.436812] alg: No test for adler32 (adler32-zlib) [ 1784.388750] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 1784.521935] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1786.151436] Key type lgssc registered [ 1787.074326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1808.967666] Key type lgssc unregistered [ 1809.293809] LNet: 98771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1809.309855] LNetError: 98771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1809.327327] LNet: Removed LNI 192.168.206.10@tcp [ 1810.103482] Key type .llcrypt unregistered [ 1810.105588] Key type ._llcrypt unregistered [ 1824.175335] Key type ._llcrypt registered [ 1824.179405] Key type .llcrypt registered [ 1825.524476] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:42:05 (1787848925) [ 1826.007279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1826.017952] alg: No test for adler32 (adler32-zlib) [ 1826.978442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.048366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1827.089233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1827.150188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1827.206937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1833.515597] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:42:13 (1787848933) [ 1833.625527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1833.665082] LNet: 100181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.714379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1833.825597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1833.869167] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1834.944246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.983916] LNet: 100331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.993256] LNetError: 100331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.012240] LNet: Removed LNI 192.168.206.10@tcp [ 1835.065836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1835.152609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1835.185465] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1835.978304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1836.001233] LNet: 100359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.008919] LNetError: 100359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1836.023958] LNet: Removed LNI 192.168.206.10@tcp [ 1836.042746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.107901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1836.129333] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1843.357632] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:42:23 (1787848943) [ 1843.503934] LNet: 100946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1843.515946] LNetError: 100946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.543662] LNet: Removed LNI 192.168.206.10@tcp [ 1844.363083] Key type .llcrypt unregistered [ 1844.368084] Key type ._llcrypt unregistered [ 1851.055752] Key type ._llcrypt registered [ 1851.058925] Key type .llcrypt registered [ 1851.344561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1851.357606] alg: No test for adler32 (adler32-zlib) [ 1852.290085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.364406] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1852.366914] LNet: *** cfs_fail_loc=e003, val=100*** [ 1853.054377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1853.070664] LNet: *** cfs_fail_loc=e003, val=100*** [ 1853.074568] LNet: Skipped 2 previous similar messages [ 1853.091535] Ping reply received with mapping: [ 1853.095918] found metadata reply NI[192.168.206.10@tcp] [ 1853.099235] 0a ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1853.102322] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1853.107619] 5a 5a 5a 5a 5a 5a 5a 5a [ 1853.245207] LNet: 101353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1853.252515] LNetError: 101353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1853.268971] LNet: Removed LNI 192.168.206.10@tcp [ 1853.792444] Key type .llcrypt unregistered [ 1853.796251] Key type ._llcrypt unregistered [ 1865.609518] Key type ._llcrypt registered [ 1865.612851] Key type .llcrypt registered [ 1866.776540] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:42:46 (1787848966) [ 1867.357356] Key type .llcrypt unregistered [ 1867.360633] Key type ._llcrypt unregistered [ 1873.681617] Key type ._llcrypt registered [ 1873.684906] Key type .llcrypt registered [ 1873.838232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1873.847688] alg: No test for adler32 (adler32-zlib) [ 1874.939164] LNetError: 102411:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1875.398335] Key type .llcrypt unregistered [ 1875.407422] Key type ._llcrypt unregistered [ 1880.694565] Key type ._llcrypt registered [ 1880.696931] Key type .llcrypt registered [ 1882.059403] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:43:01 (1787848981) [ 1882.579841] Key type .llcrypt unregistered [ 1882.583175] Key type ._llcrypt unregistered [ 1888.656291] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1888.659973] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1891.194231] Key type ._llcrypt registered [ 1891.196974] Key type .llcrypt registered [ 1891.344757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1891.363805] alg: No test for adler32 (adler32-zlib) [ 1892.296443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1892.378575] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1901.411175] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:43:21 (1787849001) [ 1901.597912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1901.632226] LNet: 104200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.636839] LNetError: 104200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1901.652289] LNet: Removed LNI 10.1.2.3@tcp [ 1901.681621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1902.454477] Key type .llcrypt unregistered [ 1902.456774] Key type ._llcrypt unregistered [ 1909.166187] Key type ._llcrypt registered [ 1909.167667] Key type .llcrypt registered [ 1909.524368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1909.557970] alg: No test for adler32 (adler32-zlib) [ 1910.458853] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1910.532763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1910.625459] LNet: 104493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.631244] LNetError: 104493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.654305] LNet: Removed LNI 192.168.206.10@tcp [ 1912.362363] Key type .llcrypt unregistered [ 1912.369374] Key type ._llcrypt unregistered [ 1924.430230] Key type ._llcrypt registered [ 1924.431720] Key type .llcrypt registered [ 1925.682308] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:43:45 (1787849025) [ 1926.415386] Key type .llcrypt unregistered [ 1926.419977] Key type ._llcrypt unregistered [ 1932.754439] Key type ._llcrypt registered [ 1932.759594] Key type .llcrypt registered [ 1933.036868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1933.048465] alg: No test for adler32 (adler32-zlib) [ 1940.800268] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:44:00 (1787849040) [ 1941.147186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1941.173725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.266437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1941.320645] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1942.106712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1942.169562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1942.220990] LNet: Removed LNI 192.168.206.10@tcp [ 1942.578502] LNet: 106250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1942.893527] Key type .llcrypt unregistered [ 1942.897294] Key type ._llcrypt unregistered [ 1952.890263] Key type ._llcrypt registered [ 1952.892379] Key type .llcrypt registered [ 1953.943422] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:44:13 (1787849053) [ 1954.406348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1954.428699] alg: No test for adler32 (adler32-zlib) [ 1955.381544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1955.430970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1955.449277] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1956.206215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1956.241604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.10@tcp [ 1956.262521] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1956.268906] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1956.271603] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1956, 0, 0 [ 1956.275914] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1956.278446] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1956.282201] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1956.285327] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1956, 0, 0 [ 1956.289063] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1956.291410] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1956.296431] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1956.305276] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1956, 0, 2006 [ 1956.312279] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1956.315243] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1956.322634] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1956.333534] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1956, 0, 2006 [ 1956.342546] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1962.041682] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:44:22 (1787849062) [ 1962.675171] LNet: 107790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.680992] LNetError: 107790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1962.697412] LNet: Removed LNI 192.168.206.10@tcp [ 1963.161376] Key type .llcrypt unregistered [ 1963.166554] Key type ._llcrypt unregistered [ 1968.782975] Key type ._llcrypt registered [ 1968.787525] Key type .llcrypt registered [ 1968.969942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1968.983438] alg: No test for adler32 (adler32-zlib) [ 1969.856628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1969.882915] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1977.578233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1987.658302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.713317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1987.737470] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1988.673907] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1988.852086] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1988.864264] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1988.869546] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1991.202766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1991.218980] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2003.143672] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:45:02 (1787849102) [ 2003.499667] LNet: 109299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.514029] LNetError: 109299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2003.522631] LNet: Removed LNI 192.168.206.10@tcp [ 2004.575195] LNet: Removed LNI 192.168.206.10@tcp1 [ 2004.580306] LNetError: 109299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2004.592358] LNetError: 109299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2004.618246] LNet: Removed LNI 10.1.2.3@tcp99 [ 2005.598899] Key type .llcrypt unregistered [ 2005.601310] Key type ._llcrypt unregistered [ 2012.579158] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2012.584062] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2015.350280] Key type ._llcrypt registered [ 2015.353599] Key type .llcrypt registered [ 2015.616188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2015.639272] alg: No test for adler32 (adler32-zlib) [ 2016.637588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2016.744220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2016.784313] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2017.835901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2017.857765] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2020.340930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2020.392411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2020.461714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2020.486940] LNetError: 110049:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2020.545837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2020.592454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2021.111179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2021.186281] LNet: 110069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2021.191651] LNetError: 110069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2021.198645] LNet: Removed LNI 10.1.2.3@tcp [ 2022.273147] LNet: Removed LNI 192.168.206.10@tcp [ 2022.965990] Key type .llcrypt unregistered [ 2022.971071] Key type ._llcrypt unregistered [ 2036.005079] Key type ._llcrypt registered [ 2036.007678] Key type .llcrypt registered [ 2037.463854] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:45:37 (1787849137) [ 2037.878535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2037.890800] alg: No test for adler32 (adler32-zlib) [ 2038.824313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.886906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2038.918180] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2039.598911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 2039.676404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg610-client@tcp [ 2045.363722] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:45:45 (1787849145) [ 2045.429862] LNet: 111600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.435895] LNetError: 111600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2045.468334] LNet: Removed LNI 192.168.206.10@tcp [ 2046.071469] Key type .llcrypt unregistered [ 2046.074371] Key type ._llcrypt unregistered [ 2051.810038] Key type ._llcrypt registered [ 2051.812265] Key type .llcrypt registered [ 2051.988700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2052.005169] alg: No test for adler32 (adler32-zlib) [ 2052.895937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2052.920031] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2052.943486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2058.472243] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:45:58 (1787849158) [ 2060.129548] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2061.822808] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:46:01 (1787849161) [ 2063.148685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2065.066225] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:46:04 (1787849164) [ 2065.276357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2065.307878] LNet: 113031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2065.314735] LNetError: 113031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2065.333227] LNet: Removed LNI 192.168.206.10@tcp [ 2065.355594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2065.416687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2065.438216] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2066.198626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2066.259720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2073.193942] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:46:12 (1787849172) [ 2073.365485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2073.407863] LNet: 113741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.424942] LNetError: 113741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2073.454203] LNet: Removed LNI 192.168.206.10@tcp [ 2073.503675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2073.581139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2073.650865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2074.426281] Key type .llcrypt unregistered [ 2074.433865] Key type ._llcrypt unregistered [ 2086.365497] Key type ._llcrypt registered [ 2086.371768] Key type .llcrypt registered [ 2087.641881] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:46:27 (1787849187) [ 2087.980101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2088.004436] alg: No test for adler32 (adler32-zlib) [ 2088.885217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.944987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2088.968673] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2089.696850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2089.748853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2096.819688] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:46:36 (1787849196) [ 2097.045155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2097.064745] LNet: 115300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2097.070799] LNetError: 115300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2097.087286] LNet: Removed LNI 192.168.206.10@tcp [ 2097.116681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.168970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2097.197717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2106.494630] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:46:45 (1787849205) [ 2106.742880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2106.774648] LNet: 115881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.796765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.874449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2107.359249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2107.402930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2107.454433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2107.987497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2108.046614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2108.198877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2108.246989] LNet: 115915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.279308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.374775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2108.828592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2108.868980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2108.912956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2109.514433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2109.571875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2116.198297] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:46:55 (1787849215) [ 2116.284542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2116.328866] LNet: 116507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2116.370318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.450509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2116.507276] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2117.325216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2117.339436] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2118.146248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2118.198502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2118.470846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.499603] LNetError: 116796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2118.503929] LNet: Removed LNI 192.168.206.10@tcp [ 2118.515606] LNet: Removed LNI 192.168.206.10@tcp2 [ 2118.539520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2118.590517] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2118.593772] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2118.598469] LNet: 116802:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2118.980130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2119.144776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2120.163067] LNet: Removed LNI 192.168.206.10@tcp [ 2120.166851] LNetError: 116839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2120.179617] LNetError: 116839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2120.204800] LNet: Removed LNI 192.168.206.10@tcp2 [ 2120.238602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.305204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2120.340692] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2121.016386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2121.039926] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2121.840731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2121.882083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2121.939784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2122.207733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2122.232345] LNet: 117123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.235636] LNet: 117123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2122.243120] LNetError: 117123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2122.248193] LNet: Removed LNI 192.168.206.10@tcp [ 2123.303423] LNet: Removed LNI 192.168.206.10@tcp2 [ 2123.324758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.371037] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2123.377839] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2123.382444] LNet: 117130:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2123.906202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2124.124959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2124.150541] LNet: Removed LNI 192.168.206.10@tcp [ 2125.222720] LNet: Removed LNI 192.168.206.10@tcp2 [ 2125.265900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2125.337203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2125.361477] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2126.067906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2126.107645] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2126.743616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2126.786325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2126.965710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2126.983728] LNetError: 117446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2126.988284] LNetError: 117446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2126.992547] LNet: Removed LNI 192.168.206.10@tcp [ 2127.007700] LNet: Removed LNI 192.168.206.10@tcp2 [ 2127.026042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.082375] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2127.086571] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2127.093294] LNet: 117452:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2127.694117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2127.853827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2128.927235] LNet: Removed LNI 192.168.206.10@tcp [ 2128.938954] LNet: Removed LNI 192.168.206.10@tcp2 [ 2128.982341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.049725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2129.074745] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2129.709731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2129.738725] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2130.493979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2130.526601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2130.636642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2130.918614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2130.942545] LNet: 117774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2130.949803] LNet: 117774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2131.999240] LNet: Removed LNI 192.168.206.10@tcp [ 2133.030445] LNet: Removed LNI 192.168.206.10@tcp2 [ 2133.062878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.143594] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2133.152480] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2133.157863] LNet: 117781:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2133.684388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2133.885653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2133.900608] LNet: Removed LNI 192.168.206.10@tcp [ 2134.956154] LNet: Removed LNI 192.168.206.10@tcp2 [ 2134.988183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.076989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2135.091943] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2135.664475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2135.699398] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2136.491719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2136.531624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2136.593724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2137.032653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.084725] LNetError: 118114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.096212] LNetError: 118114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2137.102350] LNet: Removed LNI 192.168.206.10@tcp [ 2137.112246] LNet: Removed LNI 192.168.206.10@tcp2 [ 2137.129528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.222981] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2137.230052] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2137.234591] LNet: 118120:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2137.572559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2137.856885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2138.911164] LNet: Removed LNI 192.168.206.10@tcp [ 2138.920378] LNet: Removed LNI 192.168.206.10@tcp2 [ 2138.961756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.033960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2139.058499] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2139.723536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2139.746372] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2140.733175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2140.767908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2140.800551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2140.870339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2141.147643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2142.239185] LNet: Removed LNI 192.168.206.10@tcp [ 2143.272712] LNet: Removed LNI 192.168.206.10@tcp2 [ 2143.300275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.393197] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2143.409132] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2143.703200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2150.227691] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2152.579127] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:47:31 (1787849251) [ 2152.725786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2152.770496] LNet: 119235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.779953] LNet: 119235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2152.803894] LNet: Removed LNI 192.168.206.10@tcp [ 2153.833411] LNet: Removed LNI 192.168.206.10@tcp2 [ 2153.851391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.189490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2154.238929] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2154.868992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2154.893310] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2156.408924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2156.436759] LNetError: 119515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2156.445625] LNetError: 119515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2156.449938] LNet: Removed LNI 10.1.2.3@tcp [ 2156.460815] LNet: Removed LNI 192.168.206.10@tcp [ 2156.488908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.571606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2156.607772] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2156.613631] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2157.244883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2157.279971] LNet: Removed LNI 192.168.206.10@tcp [ 2157.297603] LNet: Removed LNI 10.1.2.3@tcp [ 2165.097592] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 12:47:44 (1787849264) [ 2165.600480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2165.712370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.800740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2165.835360] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2167.359562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2167.427918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 2167.479094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 2167.542823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 2167.594087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 2167.653123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 2167.707842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 2167.784821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 2167.825642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 2167.884504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 2167.935651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 2168.104589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2176.774053] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:47:55 (1787849275) [ 2176.889461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2177.958630] LNet: Removed LNI 192.168.206.10@tcp [ 2178.003804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2178.246200] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2178.506527] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2178.509179] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2178.512214] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2180.995268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2181.037060] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2182.278955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2182.307716] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2184.180289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2184.213461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2184.265776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.10@tcp -m PUT -e local_timeout -r 1 [ 2184.322266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2184.390455] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2184.392452] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2184.717493] LNetError: 120916:0:(lib-move.c:4801:lnet_parse()) 192.168.206.10@tcp, src 192.168.206.10@tcp: Dropping PUT (error -108 looking up sender) [ 2185.440366] LNet: 121379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2185.447274] LNet: 121379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2185.465268] LNet: Removed LNI 10.1.2.3@tcp [ 2186.536563] LNet: Removed LNI 192.168.206.10@tcp [ 2187.098269] Key type .llcrypt unregistered [ 2187.101405] Key type ._llcrypt unregistered [ 2199.684942] Key type ._llcrypt registered [ 2199.689249] Key type .llcrypt registered [ 2200.870694] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:48:20 (1787849300) [ 2201.303667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2201.329100] alg: No test for adler32 (adler32-zlib) [ 2202.249695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2202.568223] LNet: 122232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.200481] Key type .llcrypt unregistered [ 2203.203845] Key type ._llcrypt unregistered [ 2208.588701] Key type ._llcrypt registered [ 2208.591121] Key type .llcrypt registered [ 2208.792153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2208.806610] alg: No test for adler32 (adler32-zlib) [ 2209.764698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2209.807631] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2216.808208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 2217.031058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2217.651029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.110@tcp [ 2217.914853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2217.952613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2217.985747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2218.015169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2218.016265] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.110@tcp [ 2218.056987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2218.083399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2218.123275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2218.191648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2218.212701] LNet: 122685:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2218.243648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2218.280321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2218.323751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2218.356960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2218.435313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2218.500984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2219.059341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2219.309458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2219.345957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2219.378742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2219.407567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2219.436253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2219.470915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2219.508597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2219.540270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2219.574345] LNet: 122802:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2219.605785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2219.671866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2219.722651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2219.754305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2219.782807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2219.824185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2220.413789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.110@tcp [ 2220.617761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2220.654784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2220.679409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2220.717506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2220.741664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2220.769155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2220.811561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2220.839133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2220.862385] LNet: 122920:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2220.887994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2220.918601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2220.951972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2220.982904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2221.011887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2221.057613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2221.670149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2221.872517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2221.898252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2221.922963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2221.949304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2221.974828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2222.010651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2222.057692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2222.086994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2222.112991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2222.133937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2222.153683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2222.175584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2222.200596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2222.223962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2222.543957] LNet: 123087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.551762] LNetError: 123087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.570250] LNet: Removed LNI 192.168.206.10@tcp [ 2222.942973] Key type .llcrypt unregistered [ 2222.945559] Key type ._llcrypt unregistered [ 2237.891407] Key type ._llcrypt registered [ 2237.895826] Key type .llcrypt registered [ 2239.062341] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:48:58 (1787849338) [ 2239.525754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2239.544448] alg: No test for adler32 (adler32-zlib) [ 2240.469486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2240.935741] LNet: 123975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2241.604638] Key type .llcrypt unregistered [ 2241.607523] Key type ._llcrypt unregistered [ 2247.667671] Key type ._llcrypt registered [ 2247.673343] Key type .llcrypt registered [ 2247.913119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2247.927654] alg: No test for adler32 (adler32-zlib) [ 2248.827535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2248.856995] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2256.236376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 2263.432322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.471674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2263.486046] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 2264.519720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2265.199850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.110@tcp [ 2265.247100] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.110@tcp [ 2265.493581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2265.534891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2265.558681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2265.609104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2265.670312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2265.722161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2265.749717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2265.791946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2265.814916] LNet: 124631:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2265.835513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2265.883423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2265.911559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2265.957574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2265.989972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2266.015641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2266.589923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2266.818378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2266.873387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2266.909795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2266.938740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2266.967290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2266.995968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2267.015726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2267.044432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2267.057746] LNet: 124749:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2267.069095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2267.089379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2267.120955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2267.156299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2267.195978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2267.223416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2268.025803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.110@tcp [ 2268.450307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2268.517208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2268.549368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2268.610575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2268.645712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2268.692098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2268.747968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2268.786159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2268.830732] LNet: 124866:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2268.859558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2268.906415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2268.956318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2269.015630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2269.059557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2269.105471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2269.879678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2270.285356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2270.350480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2270.415897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2270.491329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2270.558129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2270.657043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2270.701140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2270.782941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2270.844725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2270.899345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2270.967987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2271.029912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2271.097914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2271.169512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2277.293483] LNet: 125069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2277.297478] LNetError: 125069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2278.306296] LNet: Removed LNI 192.168.206.10@tcp [ 2278.310157] LNetError: 125069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2278.361100] LNet: Removed LNI 192.168.206.10@tcp1 [ 2278.923776] Key type .llcrypt unregistered [ 2278.926794] Key type ._llcrypt unregistered [ 2294.737177] Key type ._llcrypt registered [ 2294.739584] Key type .llcrypt registered [ 2295.827900] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 12:49:55 (1787849395) [ 2296.227886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2296.237900] alg: No test for adler32 (adler32-zlib) [ 2297.103058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.162613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2297.188416] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2297.875311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2297.891468] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2298.650422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2298.751026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2299.807727] LNet: Removed LNI 192.168.206.10@tcp [ 2305.716443] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 12:50:05 (1787849405) [ 2306.946491] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2308.587261] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:50:08 (1787849408) [ 2314.710582] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:50:14 (1787849414) [ 2321.422387] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:50:21 (1787849421) [ 2328.454881] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:50:28 (1787849428) [ 2335.191703] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:50:34 (1787849434) [ 2342.662777] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:50:42 (1787849442) [ 2349.743934] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:50:49 (1787849449) [ 2356.238886] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:50:56 (1787849456) [ 2363.178438] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:51:03 (1787849463) [ 2370.410533] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:51:09 (1787849469) [ 2376.689637] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:51:16 (1787849476) [ 2383.618502] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:51:23 (1787849483) [ 2389.804785] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:51:29 (1787849489) [ 2395.987379] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:51:35 (1787849495) [ 2404.156156] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:51:42 (1787849502) [ 2416.699219] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:51:55 (1787849515) [ 2427.138511] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:52:06 (1787849526) [ 2435.583382] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:52:15 (1787849535) [ 2435.700835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2435.759697] LNet: 136758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2435.765593] LNetError: 136758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2435.795061] LNet: Removed LNI 192.168.206.10@tcp2 [ 2435.835168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2435.989435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2436.011843] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2437.088748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2437.118696] LNet: 136908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2437.124647] LNetError: 136908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2437.148666] LNet: Removed LNI 192.168.206.10@tcp [ 2437.172965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2437.215694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2437.245937] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2443.739677] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:52:23 (1787849543) [ 2451.621889] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:52:31 (1787849551) [ 2458.656829] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:52:38 (1787849558) [ 2466.523424] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:52:46 (1787849566) [ 2473.088650] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:52:53 (1787849573) [ 2478.768483] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:52:58 (1787849578) [ 2485.288718] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:53:05 (1787849585) [ 2492.152627] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:53:11 (1787849591) [ 2500.533576] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:53:18 (1787849598) [ 2507.612873] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:53:27 (1787849607) [ 2513.950518] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:53:33 (1787849613) [ 2520.157197] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:53:40 (1787849620) [ 2526.567033] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2266 sec ========== 12:53:46 (1787849626) [ 2526.957103] LNet: 144377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2526.966162] LNetError: 144377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2526.993329] LNet: Removed LNI 192.168.206.10@tcp [ 2527.655561] Key type .llcrypt unregistered [ 2527.658052] Key type ._llcrypt unregistered [ 2538.731432] Key type ._llcrypt registered [ 2538.734254] Key type .llcrypt registered [ 2539.206605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2539.226223] alg: No test for adler32 (adler32-zlib) [ 2540.303465] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 2540.484336] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2542.161469] Key type lgssc registered [ 2543.516951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2603.416512] Lustre: Mounted lustre-client - version 2.17.57_87_ge30c52f [ 2608.017242] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2614.370458] Lustre: Unmounted lustre-client [ 2642.613957] Key type lgssc unregistered [ 2642.931786] LNet: 147341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2642.943930] LNetError: 147341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2642.967612] LNet: Removed LNI 192.168.206.10@tcp [ 2643.881294] Key type .llcrypt unregistered [ 2643.887506] Key type ._llcrypt unregistered