[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 482370579 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003236] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007451] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010155] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.013003] SELinux: Initializing. [ 0.014077] *** VALIDATE selinux *** [ 0.022384] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026524] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] *** VALIDATE tmpfs *** [ 0.030493] *** VALIDATE proc *** [ 0.031262] *** VALIDATE cgroup *** [ 0.032008] *** VALIDATE cgroup2 *** [ 0.033268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034137] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036031] Spectre V2 : User space: Vulnerable [ 0.037009] Speculative Store Bypass: Vulnerable [ 0.040584] debug: unmapping init [mem 0xffffffff90259000-0xffffffff90260fff] [ 0.042189] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043776] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044028] ... version: 2 [ 0.045015] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.049994] ... event mask: 000000070000000f [ 0.050309] rcu: Hierarchical SRCU implementation. [ 0.052514] smp: Bringing up secondary CPUs ... [ 0.053564] x86: Booting SMP configuration: [ 0.054023] .... node #0, CPUs: #1 #2 #3 [ 0.058011] smp: Brought up 1 node, 4 CPUs [ 0.060011] smpboot: Max logical packages: 1 [ 0.061015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148024] node 0 deferred pages initialised in 85ms [ 0.151217] devtmpfs: initialized [ 0.153296] x86/mm: Memory block size: 128MB [ 0.156821] gcov: version magic: 0x41383552 [ 0.159138] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.164325] pinctrl core: initialized pinctrl subsystem [ 0.166183] [ 0.166783] ************************************************************* [ 0.169013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171012] ** ** [ 0.173015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175013] ** ** [ 0.178014] ** This means that this kernel is built to expose internal ** [ 0.179010] ** IOMMU data structures, which may compromise security on ** [ 0.181014] ** your system. ** [ 0.184012] ** ** [ 0.186012] ** If you see this message and you are not debugging the ** [ 0.188025] ** kernel, report this immediately to your vendor! ** [ 0.190016] ** ** [ 0.193016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195014] ************************************************************* [ 0.197698] NET: Registered protocol family 16 [ 0.199440] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207123] cpuidle: using governor menu [ 0.208436] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.211663] PCI: Using configuration type 1 for base access [ 0.213136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.223055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.224020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.226061] cryptd: max_cpu_qlen set to 1000 [ 0.230246] ACPI: Added _OSI(Module Device) [ 0.231013] ACPI: Added _OSI(Processor Device) [ 0.233012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.238731] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243444] ACPI: Interpreter enabled [ 0.245062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246012] ACPI: Using IOAPIC for interrupt routing [ 0.247080] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.250380] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.261864] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.263064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.266020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.268083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273347] acpiphp: Slot [2] registered [ 0.275110] acpiphp: Slot [5] registered [ 0.276108] acpiphp: Slot [6] registered [ 0.277114] acpiphp: Slot [3] registered [ 0.279115] acpiphp: Slot [4] registered [ 0.280118] acpiphp: Slot [7] registered [ 0.282121] acpiphp: Slot [8] registered [ 0.283113] acpiphp: Slot [9] registered [ 0.285078] acpiphp: Slot [10] registered [ 0.286136] acpiphp: Slot [11] registered [ 0.288116] acpiphp: Slot [12] registered [ 0.290227] acpiphp: Slot [13] registered [ 0.291115] acpiphp: Slot [14] registered [ 0.293119] acpiphp: Slot [15] registered [ 0.294157] acpiphp: Slot [16] registered [ 0.296096] acpiphp: Slot [17] registered [ 0.297277] acpiphp: Slot [18] registered [ 0.300021] acpiphp: Slot [19] registered [ 0.301135] acpiphp: Slot [20] registered [ 0.302084] acpiphp: Slot [21] registered [ 0.303103] acpiphp: Slot [22] registered [ 0.305104] acpiphp: Slot [23] registered [ 0.306129] acpiphp: Slot [24] registered [ 0.308135] acpiphp: Slot [25] registered [ 0.309105] acpiphp: Slot [26] registered [ 0.311152] acpiphp: Slot [27] registered [ 0.312111] acpiphp: Slot [28] registered [ 0.314114] acpiphp: Slot [29] registered [ 0.315117] acpiphp: Slot [30] registered [ 0.317117] acpiphp: Slot [31] registered [ 0.318127] PCI host bridge to bus 0000:00 [ 0.320025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.324029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.330028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.332039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333271] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.335856] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.339296] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.347015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.351056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.354023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.356017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.358025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.361302] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.363701] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.366042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.368844] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.372917] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.383017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.387014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.393055] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.405023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.417019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.446025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.458599] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.465022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.472016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.481019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.487670] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.489294] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.491327] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.493336] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.495174] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.500029] iommu: Default domain type: Passthrough [ 0.501333] SCSI subsystem initialized [ 0.502000] ACPI: bus type USB registered [ 0.503146] usbcore: registered new interface driver usbfs [ 0.505114] usbcore: registered new interface driver hub [ 0.507094] usbcore: registered new device driver usb [ 0.509176] pps_core: LinuxPPS API ver. 1 registered [ 0.510010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.513056] PTP clock support registered [ 0.515042] EDAC MC: Ver: 3.0.0 [ 0.516382] PCI: Using ACPI for IRQ routing [ 0.517634] NetLabel: Initializing [ 0.519012] NetLabel: domain hash size = 128 [ 0.520009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.521061] NetLabel: unlabeled traffic allowed by default [ 0.523134] vgaarb: loaded [ 0.524275] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.526014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.531983] clocksource: Switched to clocksource kvm-clock [ 0.642980] VFS: Disk quotas dquot_6.6.0 [ 0.644704] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.647394] *** VALIDATE ramfs *** [ 0.648755] *** VALIDATE hugetlbfs *** [ 0.650418] pnp: PnP ACPI init [ 0.652894] pnp: PnP ACPI: found 6 devices [ 0.673235] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.676930] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.679391] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.681746] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.684283] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.686113] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.688621] NET: Registered protocol family 2 [ 0.691502] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.696724] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.700406] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.705482] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.708153] TCP: Hash tables configured (established 65536 bind 65536) [ 0.710664] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.713576] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.716336] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.719150] NET: Registered protocol family 1 [ 0.721571] RPC: Registered named UNIX socket transport module. [ 0.723597] RPC: Registered udp transport module. [ 0.725316] RPC: Registered tcp transport module. [ 0.727080] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.729397] NET: Registered protocol family 44 [ 0.731088] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.733239] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.735323] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.737780] PCI: CLS 0 bytes, default 64 [ 0.739369] Unpacking initramfs... [ 2.215708] debug: unmapping init [mem 0xffff95cf7cc64000-0xffff95cf7ffcffff] [ 2.222532] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.224764] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.228973] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.725659] Initialise system trusted keyrings [ 2.727178] Key type blacklist registered [ 2.729579] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.738066] zbud: loaded [ 2.741369] *** VALIDATE nfs *** [ 2.742468] *** VALIDATE nfs4 *** [ 2.743971] pstore: using deflate compression [ 2.747175] Platform Keyring initialized [ 2.881708] NET: Registered protocol family 38 [ 2.882965] Key type asymmetric registered [ 2.884137] Asymmetric key parser 'x509' registered [ 2.885483] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.887905] io scheduler mq-deadline registered [ 2.889246] io scheduler kyber registered [ 2.890413] io scheduler bfq registered [ 2.891904] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.893889] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.896253] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.898319] ACPI: Power Button [PWRF] [ 2.902624] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.909186] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.922191] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.950360] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.979394] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.985276] Non-volatile memory driver v1.3 [ 2.987307] Linux agpgart interface v0.103 [ 3.021573] virtio_blk virtio1: [vda] 146712 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.024614] vda: detected capacity change from 0 to 75116544 [ 3.039721] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.042425] vdb: detected capacity change from 0 to 1073741824 [ 3.048485] libphy: Fixed MDIO Bus: probed [ 3.063447] usbcore: registered new interface driver usbserial_generic [ 3.065682] usbserial: USB Serial support registered for generic [ 3.067553] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.071515] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.072968] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.075385] mousedev: PS/2 mouse device common for all mice [ 3.077395] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.077760] rtc_cmos 00:05: RTC can wake from S4 [ 3.082436] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.082693] rtc_cmos 00:05: registered as rtc0 [ 3.087905] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.088649] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.093987] intel_pstate: CPU model not supported [ 3.097675] hid: raw HID events driver (C) Jiri Kosina [ 3.100250] usbcore: registered new interface driver usbhid [ 3.102687] usbhid: USB HID core driver [ 3.105474] drop_monitor: Initializing network drop monitor service [ 3.108424] Initializing XFRM netlink socket [ 3.112896] NET: Registered protocol family 10 [ 3.116093] Segment Routing with IPv6 [ 3.117639] NET: Registered protocol family 17 [ 3.120643] mpls_gso: MPLS GSO support [ 3.126306] RAS: Correctable Errors collector initialized. [ 3.128484] AVX version of gcm_enc/dec engaged. [ 3.130459] AES CTR mode by8 optimization enabled [ 3.219900] sched_clock: Marking stable (3219866510, 0)->(4111378978, -891512468) [ 3.223369] registered taskstats version 1 [ 3.226273] Loading compiled-in X.509 certificates [ 3.228227] zswap: loaded using pool lzo/zbud [ 3.256589] Key type big_key registered [ 3.270304] Key type encrypted registered [ 3.272056] ima: No TPM chip found, activating TPM-bypass! [ 3.274222] ima: Allocated hash algorithm: sha1 [ 3.275840] ima: No architecture policies found [ 3.277131] evm: Initialising EVM extended attributes: [ 3.278930] evm: security.selinux [ 3.280081] evm: security.ima [ 3.281257] evm: security.capability [ 3.282406] evm: HMAC attrs: 0x1 [ 3.284242] rtc_cmos 00:05: setting system clock to 2026-09-08 04:57:19 UTC (1788843439) [ 3.290377] debug: unmapping init [mem 0xffffffff91203000-0xffffffff913fffff] [ 3.293175] debug: unmapping init [mem 0xffffffff8ff82000-0xffffffff90258fff] [ 3.300102] Write protecting the kernel read-only data: 28672k [ 3.303477] debug: unmapping init [mem 0xffffffff8e603000-0xffffffff8e7fffff] [ 3.305990] debug: unmapping init [mem 0xffffffff8ef14000-0xffffffff8effffff] [ 3.364249] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.371835] systemd[1]: Detected virtualization kvm. [ 3.373864] systemd[1]: Detected architecture x86-64. [ 3.375760] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.400448] systemd[1]: No hostname configured. [ 3.402141] systemd[1]: Set hostname to . [ 3.403970] random: systemd: uninitialized urandom read (16 bytes read) [ 3.406367] systemd[1]: Initializing machine ID from random generator. [ 3.457162] random: ln: uninitialized urandom read (6 bytes read) [ 3.539536] random: systemd: uninitialized urandom read (16 bytes read) [ 3.542596] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.548288] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.553512] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.133481] device-mapper: uevent: version 1.0.3 [ 4.136892] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] [ 4.727700] random: fast init done Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook...[ 4.770547] virtio_net virtio0 ens2: renamed from eth0 [ 4.930917] scsi host0: ata_piix [ 4.950336] scsi host1: ata_piix [ 4.952478] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.954274] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.463308] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.787925] random: crng init done [ 9.789441] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.120536] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.168781] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.405836] SELinux: Disabled at runtime. [ 11.464699] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.473559] systemd[1]: Detected virtualization kvm. [ 11.474930] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.952741] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.955131] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.958984] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.961879] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.966218] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.972879] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.976578] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ 12.062274] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.358127] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.649072] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.677324] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.752221] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.766864] EDAC sbridge: Ver: 1.1.2 [ 13.723459] Key type dns_resolver registered [ 14.026470] NFS: Registering the id_resolver key type [ 14.028717] Key type id_resolver registered [ 14.030438] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg229-client login: [ 43.511690] libcfs: loading out-of-tree module taints kernel. [ 43.574146] Key type ._llcrypt registered [ 43.575894] Key type .llcrypt registered [ 43.862280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 43.869261] alg: No test for adler32 (adler32-zlib) [ 44.876234] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 45.275702] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 46.936252] Key type lgssc registered [ 49.691515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.633023] hrtimer: interrupt took 2060830 ns [ 204.081686] Lustre: Mounted lustre-client - version 2.17.57_111_g33f3974 [ 209.771649] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 224.715525] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing check_logdir /tmp/testlogs/ [ 229.861265] Lustre: lustre-OST0000-osc-ffff95cfc99df800: disconnect after 23s idle [ 230.316749] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing yml_node [ 236.351697] Lustre: DEBUG MARKER: Client: 2.17.57.111 [ 239.275842] Lustre: DEBUG MARKER: MDS: 2.17.57.111 [ 241.965033] Lustre: DEBUG MARKER: OSS: 2.17.57.111 [ 244.161355] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 8 01:01:18 EDT 2026 [ 261.834576] Lustre: Unmounted lustre-client [ 289.861848] Key type lgssc unregistered [ 290.170402] LNet: 8431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.178791] LNetError: 8431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 290.205024] LNet: Removed LNI 192.168.202.29@tcp [ 291.121271] Key type .llcrypt unregistered [ 291.124572] Key type ._llcrypt unregistered [ 322.036217] Key type ._llcrypt registered [ 322.039284] Key type .llcrypt registered [ 322.413603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 322.446579] alg: No test for adler32 (adler32-zlib) [ 323.375621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 323.467750] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 323.492383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 328.535589] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 328.624249] LNet: 9302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.640921] LNetError: 9302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 328.660815] LNet: Removed LNI 192.168.202.29@tcp [ 329.379392] Key type .llcrypt unregistered [ 329.381236] Key type ._llcrypt unregistered [ 336.542919] Key type ._llcrypt registered [ 336.545643] Key type .llcrypt registered [ 338.290155] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 01:02:52 (1788843772) [ 338.680558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 338.697230] alg: No test for adler32 (adler32-zlib) [ 339.668398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.754329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.773927] LNet: 9720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.794601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.882655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 339.932598] LNet: 9727:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 347.987196] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 01:03:02 (1788843782) [ 348.110730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.169331] LNet: 10300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.228253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.400888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 355.343650] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 01:03:09 (1788843789) [ 355.452520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.497244] LNet: 10886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.554702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.687872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 363.204937] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 01:03:17 (1788843797) [ 363.508291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.536862] LNet: 11478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.551553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.608349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 370.168894] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 01:03:24 (1788843804) [ 370.280434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.321749] LNet: 12057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.358092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.426605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 370.703326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 377.579876] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 01:03:32 (1788843812) [ 377.845695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.908533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.990522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 378.373765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 378.784360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 385.733191] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 01:03:40 (1788843820) [ 386.061579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.099598] LNet: 13252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.120263] LNet: 13252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 386.143599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.222464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 393.225409] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 01:03:48 (1788843828) [ 393.421548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.495169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.552449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 393.586884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 394.113167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 394.144699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 394.600123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 394.639824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 395.102933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 395.141469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 395.583835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 395.615652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 396.159444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 396.229329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 396.591398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 396.631332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 397.037877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 397.060612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 397.452426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 397.505410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 397.939328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 397.992624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 398.541331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 398.582181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 399.081904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 399.119113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 399.634800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 399.671392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 407.045438] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 01:04:01 (1788843841) [ 407.315333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.345980] LNet: 14519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.350469] LNet: 14519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 407.368576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.428825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 407.459449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 415.652051] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 01:04:10 (1788843850) [ 416.074854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.139243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.217363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 416.282666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 422.358216] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 01:04:17 (1788843857) [ 422.703326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.740887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.813944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 422.852866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 429.882804] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 01:04:24 (1788843864) [ 430.218585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.261303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.339894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 430.390410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 437.147580] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 01:04:32 (1788843872) [ 437.351893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.377935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.428980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 437.463501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 444.827348] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 01:04:39 (1788843879) [ 445.009614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.055120] LNet: 17448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 445.068076] LNet: 17448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 445.115357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.213574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 445.283960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 453.018961] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 01:04:47 (1788843887) [ 453.177021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.242565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.355595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 453.936446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.758619] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 01:04:55 (1788843895) [ 461.039329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.127661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.358145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.614782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 469.045817] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 01:05:03 (1788843903) [ 469.249553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.317673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.461621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.638699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 478.234131] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 01:05:12 (1788843912) [ 478.355141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.447204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.084717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 479.427908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 486.501898] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 01:05:21 (1788843921) [ 486.822881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.880759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.965162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 487.025323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 493.928483] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 01:05:28 (1788843928) [ 494.042865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.079366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.152840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 494.192682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 502.041913] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 01:05:36 (1788843936) [ 502.350863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.414830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.531709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 512.238737] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 01:05:46 (1788843946) [ 512.336715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.376660] LNet: 22300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.386896] LNet: 22300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 512.432965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.533385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 522.179587] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 01:05:56 (1788843956) [ 522.466391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.536849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.633446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 523.127471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 531.029689] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 01:06:05 (1788843965) [ 531.183910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.252082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.416804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 531.472196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 539.908623] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:06:14 (1788843974) [ 540.020343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.128350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.271784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 540.315429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 548.202267] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:06:22 (1788843982) [ 548.302846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.377679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.517950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 548.574625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 556.297876] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:06:30 (1788843990) [ 556.458815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.538096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.644795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 556.680132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 564.177672] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:06:38 (1788843998) [ 564.292549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.366463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.449300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 564.579353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 564.609900] LNetError: 25844:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 564.698289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 564.764583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 572.873676] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:06:47 (1788844007) [ 573.050278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.166981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.316157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.QsARuPkY2w [ 573.452842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.QsARuPkY2w [ 573.525820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.QsARuPkY2w [ 582.693610] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:06:56 (1788844016) [ 583.291776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.361697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.475588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 583.530038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 583.605600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 591.837853] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 01:07:06 (1788844026) [ 592.005840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.083689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.148969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 592.176505] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 593.362687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 594.953672] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 596.960185] Key type lgssc registered [ 598.270893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 601.573570] LNetError: 27606:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.29@tcp99 has no available nets [ 604.052664] Key type lgssc unregistered [ 604.404840] LNetError: 28712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 605.480750] LNet: Removed LNI 192.168.202.29@tcp [ 606.317986] Key type .llcrypt unregistered [ 606.329119] Key type ._llcrypt unregistered [ 612.561287] Key type ._llcrypt registered [ 612.565718] Key type .llcrypt registered [ 613.994432] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:07:28 (1788844048) [ 614.545493] Key type .llcrypt unregistered [ 614.548390] Key type ._llcrypt unregistered [ 620.869751] Key type ._llcrypt registered [ 620.873333] Key type .llcrypt registered [ 620.981829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 620.996030] alg: No test for adler32 (adler32-zlib) [ 621.892102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.964312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 621.979919] LNet: 29622:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 621.984879] LNet: 29622:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 621.990273] LNet: 29622:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 622.002171] LNet: Message forwarding will be enabled in 2s [ 625.056249] LNet: Message forwarding enabled [ 625.551844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 625.577879] LNet: Message forwarding will stop in 2s to 6s [ 629.155526] LNet: Message forwarding disabled [ 630.238947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 630.268105] LNet: 29721:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 630.284460] LNet: 29721:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 630.292968] LNet: 29721:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 630.307413] LNet: Message forwarding will be enabled in 2s [ 630.480265] LNet: 29734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 631.193037] Key type .llcrypt unregistered [ 631.194781] Key type ._llcrypt unregistered [ 643.417609] Key type ._llcrypt registered [ 643.419414] Key type .llcrypt registered [ 644.815933] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:07:59 (1788844079) [ 645.484521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 645.506719] alg: No test for adler32 (adler32-zlib) [ 646.433454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.513855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 654.209327] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:08:08 (1788844088) [ 654.430639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.479663] LNet: 31127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.531753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.668586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 654.705870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 654.733391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 654.795439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 654.846732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 654.875546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 654.909846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 654.942099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 654.983635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 655.024935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 655.062321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 655.106247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 655.146950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 655.181479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 655.211608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 655.242768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 655.274873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 655.306840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 655.338810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 655.357378] LNetError: 31178:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 661.854885] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:08:16 (1788844096) [ 662.145757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.172177] LNet: 31749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.192262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.302816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.014531] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:08:24 (1788844104) [ 672.076892] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg229-server [ 674.190578] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:08:29 (1788844109) [ 674.369522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.418467] LNet: 32682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.456742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.535668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 674.558571] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 675.387893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[30-34]@tcp [ 675.406683] LNet: 32815:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 675.466174] LNetError: 32686:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.31@tcp is being used as a gateway but routing feature is not turned on [ 675.491160] LNetError: 32686:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.31@tcp has gone from up to down [ 681.945672] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:08:37 (1788844117) [ 683.759221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg229-server [ 686.228085] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:08:40 (1788844120) [ 686.355489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.401355] LNet: 33744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.409310] LNetError: 33744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 687.470129] LNet: Removed LNI 192.168.202.29@tcp [ 687.541538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.612821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 687.671911] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 688.417540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[30-35/2]@tcp [ 688.441739] LNet: 33870:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 688.463617] LNet: 33870:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 688.497437] LNetError: 33757:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.30@tcp is being used as a gateway but routing feature is not turned on [ 688.512303] LNetError: 33757:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 688.524362] LNetError: 33757:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.30@tcp has gone from up to down [ 688.532240] LNetError: 33757:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 688.543176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[30-35/2]@tcp [ 695.484620] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:08:50 (1788844130) [ 695.810258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.851227] LNet: 34463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.866517] LNetError: 34463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 695.895217] LNet: Removed LNI 192.168.202.29@tcp [ 695.937229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.089393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 696.179673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.233498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.322472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.400795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.443456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.491874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 696.580499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.621529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.715236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.811274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.855658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.922485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 697.023259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.064689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.169966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.287964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.324882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.379875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 697.481707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.537605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.645065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.774309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.819591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.872975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 697.940423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.977679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.042568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.111871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.142689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.185380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 698.248821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.295152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.363436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 705.415317] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:09:00 (1788844140) [ 707.268383] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg229-server [ 709.349265] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:09:04 (1788844144) [ 711.787702] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg229-server [ 713.582548] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:09:08 (1788844148) [ 713.806347] LNet: 35861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.814194] LNet: 35861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 714.489859] Key type .llcrypt unregistered [ 714.500332] Key type ._llcrypt unregistered [ 721.571591] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 721.787261] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 721.790516] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 721.796752] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 724.373379] Key type ._llcrypt registered [ 724.374927] Key type .llcrypt registered [ 724.594679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 724.618921] alg: No test for adler32 (adler32-zlib) [ 725.639344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.717534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 725.741057] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 726.690563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 726.714596] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 729.815851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 729.837723] LNet: Removed LNI 10.1.2.3@tcp [ 729.989595] LNet: 36575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.994992] LNetError: 36575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.009799] LNet: Removed LNI 192.168.202.29@tcp [ 730.851899] Key type .llcrypt unregistered [ 730.854786] Key type ._llcrypt unregistered [ 743.348542] Key type ._llcrypt registered [ 743.350791] Key type .llcrypt registered [ 744.646550] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:09:39 (1788844179) [ 745.376541] Key type .llcrypt unregistered [ 745.380241] Key type ._llcrypt unregistered [ 751.190156] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 751.414829] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 751.418537] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 751.421494] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 754.115885] Key type ._llcrypt registered [ 754.121708] Key type .llcrypt registered [ 754.379312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 754.401786] alg: No test for adler32 (adler32-zlib) [ 755.339359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.379707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 755.402521] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 756.253415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 756.283077] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 758.661555] LNet: 38064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.667176] LNetError: 38064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 758.673739] LNet: Removed LNI 10.1.2.3@tcp [ 758.687961] LNet: Removed LNI 192.168.202.29@tcp [ 759.523680] Key type .llcrypt unregistered [ 759.529740] Key type ._llcrypt unregistered [ 770.772681] Key type ._llcrypt registered [ 770.775237] Key type .llcrypt registered [ 772.240317] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:10:06 (1788844206) [ 772.737645] Key type .llcrypt unregistered [ 772.746576] Key type ._llcrypt unregistered [ 778.750427] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 779.045750] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 779.053713] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 779.058600] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 781.744385] Key type ._llcrypt registered [ 781.746563] Key type .llcrypt registered [ 781.932295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 781.964760] alg: No test for adler32 (adler32-zlib) [ 782.979392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.057681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 783.093984] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 785.690682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 785.729849] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 788.397924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 788.442433] LNet: Removed LNI 10.1.2.3@tcp [ 788.466453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 788.502470] LNet: Removed LNI 10.1.2.31@tcp [ 788.535418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 788.559356] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 789.511871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 789.543442] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 790.553904] LNet: 39980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.560482] LNetError: 39980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 790.566104] LNet: Removed LNI 10.1.2.31@tcp [ 790.580468] LNet: Removed LNI 10.1.2.3@tcp [ 791.380929] Key type .llcrypt unregistered [ 791.385089] Key type ._llcrypt unregistered [ 803.095040] Key type ._llcrypt registered [ 803.100891] Key type .llcrypt registered [ 804.314468] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:10:39 (1788844239) [ 804.938128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 804.956276] alg: No test for adler32 (adler32-zlib) [ 805.866515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.949883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 805.984477] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 806.680382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.706672] LNet: 40930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.712624] LNetError: 40930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.736637] LNet: Removed LNI 192.168.202.29@tcp [ 806.753526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.822524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.29@tcp [ 806.859708] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 807.709244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.29@tcp [ 807.739397] LNet: Removed LNI 192.168.202.29@tcp [ 807.803632] LNet: 41065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.386957] Key type .llcrypt unregistered [ 808.390463] Key type ._llcrypt unregistered [ 820.448262] Key type ._llcrypt registered [ 820.450285] Key type .llcrypt registered [ 821.622730] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:10:56 (1788844256) [ 823.457522] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg229-server [ 825.301934] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:11:00 (1788844260) [ 825.991300] Key type .llcrypt unregistered [ 825.993504] Key type ._llcrypt unregistered [ 832.494455] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 832.775254] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 832.789339] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 832.793317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 835.401261] Key type ._llcrypt registered [ 835.403257] Key type .llcrypt registered [ 835.584395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 835.595491] alg: No test for adler32 (adler32-zlib) [ 836.500241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.575247] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 837.251206] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 839.073990] LNet: 42897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.081341] LNetError: 42897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.085389] LNet: Removed LNI 192.168.202.29@tcp1 [ 839.094237] LNet: Removed LNI 10.1.2.3@tcp2 [ 839.742518] Key type .llcrypt unregistered [ 839.745279] Key type ._llcrypt unregistered [ 852.321668] Key type ._llcrypt registered [ 852.326249] Key type .llcrypt registered [ 853.635299] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:11:28 (1788844288) [ 854.234921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 854.256829] alg: No test for adler32 (adler32-zlib) [ 855.239587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.331653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 855.379831] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 856.726496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.758373] LNet: 43864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.768187] LNetError: 43864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.790703] LNet: Removed LNI 192.168.202.29@tcp [ 856.816348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.891113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 856.924357] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 865.473672] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:11:39 (1788844299) [ 865.545162] LNet: 44448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.554107] LNetError: 44448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.573315] LNet: Removed LNI 192.168.202.29@tcp [ 866.306492] Key type .llcrypt unregistered [ 866.308123] Key type ._llcrypt unregistered [ 871.393570] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 871.513990] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 871.529697] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 871.533901] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 874.280757] Key type ._llcrypt registered [ 874.284092] Key type .llcrypt registered [ 874.514686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 874.525167] alg: No test for adler32 (adler32-zlib) [ 875.428705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.488474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 875.510175] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 876.420626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 876.438812] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 879.163730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.182306] LNet: 45197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.187979] LNetError: 45197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 879.200252] LNet: Removed LNI 10.1.2.3@tcp [ 879.213801] LNet: Removed LNI 192.168.202.29@tcp [ 879.241331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.329616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 879.363475] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 879.367436] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 879.939865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.977397] LNet: 45224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.988232] LNetError: 45224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 879.993989] LNetError: 45224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 880.013687] LNet: Removed LNI 10.1.2.3@tcp [ 880.032202] LNet: Removed LNI 192.168.202.29@tcp [ 880.064592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.243355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 880.307066] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 880.314599] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 881.059645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 881.106289] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 883.110688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.141919] LNet: 45386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.151443] LNetError: 45386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 883.157450] LNetError: 45386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 883.165109] LNet: Removed LNI 10.1.2.3@tcp [ 883.169460] LNet: Removed LNI 192.168.202.29@tcp [ 883.183950] LNet: Removed LNI 192.168.202.29@tcp2 [ 883.237123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.357936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 883.394143] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 883.405862] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 883.411031] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 894.334936] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:12:08 (1788844328) [ 894.735785] LNet: 45984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.753216] LNetError: 45984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 894.761313] LNetError: 45984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 894.767695] LNet: Removed LNI 10.1.2.3@tcp [ 894.769690] LNet: Removed LNI 192.168.202.29@tcp [ 894.788348] LNet: Removed LNI 192.168.202.29@tcp2 [ 895.613441] Key type .llcrypt unregistered [ 895.617225] Key type ._llcrypt unregistered [ 902.132758] Key type ._llcrypt registered [ 902.137705] Key type .llcrypt registered [ 902.489429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.515254] alg: No test for adler32 (adler32-zlib) [ 903.465705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.507723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 903.547841] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 904.313780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.345496] LNet: 46377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.354510] LNetError: 46377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.374796] LNet: Removed LNI 192.168.202.29@tcp [ 904.423421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.528366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 904.577255] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 905.159736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.190281] LNet: 46408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.202339] LNetError: 46408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.215854] LNet: Removed LNI 192.168.202.29@tcp [ 905.237984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.304745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 905.327118] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 905.965148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.013489] LNet: Removed LNI 192.168.202.29@tcp [ 906.037371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.119850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 906.138546] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 906.760823] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 906.907459] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 906.914700] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 906.922809] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 909.473465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.492553] LNet: 46516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.499413] LNet: 46516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 909.508312] LNetError: 46516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.512685] LNetError: 46516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 909.535246] LNet: Removed LNI 192.168.202.29@tcp [ 909.577522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.651134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 909.693095] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 917.402820] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:12:31 (1788844351) [ 917.660605] LNet: 47114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.668915] LNetError: 47114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.685810] LNet: Removed LNI 192.168.202.29@tcp [ 918.275853] Key type .llcrypt unregistered [ 918.283967] Key type ._llcrypt unregistered [ 924.709280] Key type ._llcrypt registered [ 924.711626] Key type .llcrypt registered [ 924.908168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 924.921442] alg: No test for adler32 (adler32-zlib) [ 925.787823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.829937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 925.850745] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 926.568819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.584691] LNet: 47522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.587952] LNetError: 47522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.603934] LNet: Removed LNI 192.168.202.29@tcp [ 926.618373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.677136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 926.700739] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 932.994533] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:12:47 (1788844367) [ 933.165345] LNet: 48113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.176301] LNetError: 48113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.193943] LNet: Removed LNI 192.168.202.29@tcp [ 933.626623] Key type .llcrypt unregistered [ 933.630246] Key type ._llcrypt unregistered [ 939.167372] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 939.431832] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 939.446711] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 939.463770] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 942.133983] Key type ._llcrypt registered [ 942.136474] Key type .llcrypt registered [ 942.391909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 942.408812] alg: No test for adler32 (adler32-zlib) [ 943.285186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.325779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 943.342681] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 944.238543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.252861] LNet: 48561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.258699] LNetError: 48561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.271052] LNet: Removed LNI 192.168.202.29@tcp [ 944.289189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.329917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 944.352975] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 951.522544] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:13:06 (1788844386) [ 951.616345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.656114] LNet: 49154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.669152] LNetError: 49154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.712108] LNet: Removed LNI 192.168.202.29@tcp [ 951.814452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.968449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 958.571462] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:13:12 (1788844392) [ 958.663891] LNet: 49732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.401833] Key type .llcrypt unregistered [ 959.405183] Key type ._llcrypt unregistered [ 965.933798] Key type ._llcrypt registered [ 965.937532] Key type .llcrypt registered [ 966.113344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 966.133576] alg: No test for adler32 (adler32-zlib) [ 967.122026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.179664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 967.207671] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 968.101241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.133787] LNet: 50135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.138480] LNetError: 50135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.154847] LNet: Removed LNI 192.168.202.29@tcp [ 968.173762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.243903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 968.273406] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 975.300940] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:13:29 (1788844409) [ 975.403516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.422560] LNet: 50722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.427026] LNetError: 50722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.437336] LNet: Removed LNI 192.168.202.29@tcp [ 975.458030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.551568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 981.773516] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:13:36 (1788844416) [ 983.136514] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 984.446816] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:13:39 (1788844419) [ 985.765472] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 987.414463] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:13:42 (1788844422) [ 989.081480] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 991.097393] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:13:45 (1788844425) [ 992.775792] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 994.377196] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:13:49 (1788844429) [ 995.941236] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 997.581776] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:13:52 (1788844432) [ 997.856147] LNet: 52782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.437719] Key type .llcrypt unregistered [ 998.440525] Key type ._llcrypt unregistered [ 1004.917799] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1005.160496] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1005.176818] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1005.202180] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1007.792475] Key type ._llcrypt registered [ 1007.794323] Key type .llcrypt registered [ 1007.937795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1007.957377] alg: No test for adler32 (adler32-zlib) [ 1008.892810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.978755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1009.004912] LNet: Added LNI 192.168.202.29@tcp [8/512/0/180] [ 1009.009751] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1009.012500] LNet: 53100:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1009.106199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1009.126992] LNet: 53119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.135398] LNetError: 53119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1009.139949] LNet: Removed LNI 10.1.2.3@tcp [ 1009.142517] LNetError: 53119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.158487] LNet: Removed LNI 192.168.202.29@tcp [ 1009.186529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.270784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1009.291511] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1009.370035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1009.388878] LNet: 53143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.403579] LNetError: 53143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.422539] LNet: Removed LNI 192.168.202.29@tcp [ 1009.442075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.515853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1009.538249] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1009.542060] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1009.546182] LNet: 53151:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1009.649908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1009.693477] LNet: 53169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.702155] LNetError: 53169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1009.715319] LNet: Removed LNI 10.1.2.3@tcp [ 1009.716882] LNetError: 53169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.736717] LNet: Removed LNI 192.168.202.29@tcp [ 1009.767951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.837537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1009.854516] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1009.859803] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1009.862667] LNet: 53177:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1010.035888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1010.053731] LNet: 53203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.058852] LNetError: 53203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1010.063777] LNet: Removed LNI 10.1.2.3@tcp [ 1010.066739] LNetError: 53203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.077718] LNet: Removed LNI 192.168.202.29@tcp [ 1010.092154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.147585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1010.164040] LNet: Added LNI 192.168.202.29@tcp [8/256/0/182] [ 1010.168753] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1010.253860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1010.269404] LNet: 53231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.275782] LNetError: 53231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1010.280181] LNet: Removed LNI 10.1.2.3@tcp [ 1010.283558] LNetError: 53231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.297886] LNet: Removed LNI 192.168.202.29@tcp [ 1010.314700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.372816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1010.396995] LNet: Added LNI 192.168.202.29@tcp [8/256/0/182] [ 1010.403081] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1010.406757] LNet: 53239:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1010.487765] LNet: Removed LNI 10.1.2.3@tcp [ 1010.515131] LNet: Removed LNI 192.168.202.29@tcp [ 1011.051953] Key type .llcrypt unregistered [ 1011.054340] Key type ._llcrypt unregistered [ 1023.042629] Key type ._llcrypt registered [ 1023.046414] Key type .llcrypt registered [ 1024.290756] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 01:14:19 (1788844459) [ 1026.056893] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1027.788295] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:14:22 (1788844462) [ 1028.229527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1028.253877] alg: No test for adler32 (adler32-zlib) [ 1029.187186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.268564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1029.310903] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1030.282577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.29 [ 1030.295560] LNet: 54501:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1030.303554] LNet: Removed LNI 192.168.202.29@tcp [ 1036.256358] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:14:31 (1788844471) [ 1036.379454] LNet: 55065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.388150] LNetError: 55065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.402165] LNet: Removed LNI 192.168.202.29@tcp [ 1036.918473] Key type .llcrypt unregistered [ 1036.920622] Key type ._llcrypt unregistered [ 1043.193299] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1043.376522] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1046.082738] Key type ._llcrypt registered [ 1046.084385] Key type .llcrypt registered [ 1046.282547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1046.308980] alg: No test for adler32 (adler32-zlib) [ 1047.676857] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1055.899663] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:14:50 (1788844490) [ 1056.129163] LNet: 55979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.137751] LNetError: 55979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1056.190715] LNet: Removed LNI 10.1.2.3@tcp [ 1057.068681] Key type .llcrypt unregistered [ 1057.074429] Key type ._llcrypt unregistered [ 1065.957940] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1066.284996] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1066.294428] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1066.306611] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1069.430327] Key type ._llcrypt registered [ 1069.436577] Key type .llcrypt registered [ 1069.763267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1069.789821] alg: No test for adler32 (adler32-zlib) [ 1070.787847] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1078.017306] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:15:13 (1788844513) [ 1079.974119] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1081.797494] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:15:16 (1788844516) [ 1082.132155] LNet: 57162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.137439] LNetError: 57162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1082.152630] LNet: Removed LNI 10.1.2.3@tcp [ 1082.722602] Key type .llcrypt unregistered [ 1082.723984] Key type ._llcrypt unregistered [ 1089.426779] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1089.642665] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1089.647435] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1089.650109] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1092.328782] Key type ._llcrypt registered [ 1092.331510] Key type .llcrypt registered [ 1092.562027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1092.603317] alg: No test for adler32 (adler32-zlib) [ 1093.492736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1093.554395] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1101.321962] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:15:36 (1788844536) [ 1101.962869] LNet: 58388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1101.967283] LNetError: 58388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1101.981428] LNet: Removed LNI 10.1.2.3@tcp [ 1102.745593] Key type .llcrypt unregistered [ 1102.749890] Key type ._llcrypt unregistered [ 1108.583954] Key type ._llcrypt registered [ 1108.586661] Key type .llcrypt registered [ 1108.960646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1108.985934] alg: No test for adler32 (adler32-zlib) [ 1109.989676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1110.041814] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1118.257750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1118.820755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_interrupt -r 1 [ 1118.852347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_interrupt -r 1 [ 1118.880348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1118.895978] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1119.083119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.118853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1119.147328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.175714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.203715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.238985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.264534] LNet: 1 local NIs in recovery (showing 1): 192.168.202.29@tcp [ 1119.271485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1119.302865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1119.327722] LNet: 58822:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1119.350478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1119.383119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1119.432799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1119.464455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1119.500865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.535853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1119.599567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1120.189700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_dropped -r 1 [ 1120.227619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_dropped -r 1 [ 1120.266544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1120.309152] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1120.322542] LNet: Skipped 1 previous similar message [ 1120.536681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1120.563941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1120.599809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1120.639674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1120.670602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1120.715433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1120.772889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1120.805267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1120.830284] LNet: 58931:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1120.845329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.866359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1120.897615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.922545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.950055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.969921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1121.016641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1121.611298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_aborted -r 1 [ 1121.654159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_aborted -r 1 [ 1121.688037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1121.720535] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1121.931164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.966319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1121.997848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.020660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.043430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.065740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.100819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1122.140668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1122.159511] LNet: 59041:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1122.176824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.206477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.232249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.254699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.277270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.310145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.343718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1122.887751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_no_route -r 1 [ 1122.911366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_no_route -r 1 [ 1122.938603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1123.249524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1123.289263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1123.338213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1123.373590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1123.409419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1123.449353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1123.477889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1123.510566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1123.544996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1123.572865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1123.601989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1123.631923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1123.659803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.692444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1123.719896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1124.283449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_timeout -r 1 [ 1124.320395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_timeout -r 1 [ 1124.354477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1124.371428] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1124.380222] LNet: Skipped 1 previous similar message [ 1124.609650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.643918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1124.686735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.720386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.754921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.805733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.841774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.885363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.903279] LNet: 59259:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.910137] LNet: 59259:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1124.934765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.999891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1125.042927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1125.106527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1125.152686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1125.216411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1125.260243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1125.735586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_error -r 1 [ 1125.784734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_error -r 1 [ 1125.809729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1126.012444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.037565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1126.061495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.088324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.117905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.146329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.169443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.197280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1126.218534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1126.243695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1126.266953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.291629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.316694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.355348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.382389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1126.600074] LNet: 59409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1126.605980] LNetError: 59409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.657463] LNet: Removed LNI 192.168.202.29@tcp [ 1128.168162] Key type .llcrypt unregistered [ 1128.170311] Key type ._llcrypt unregistered [ 1147.589323] Key type ._llcrypt registered [ 1147.596751] Key type .llcrypt registered [ 1148.971824] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:16:23 (1788844583) [ 1149.866647] Key type .llcrypt unregistered [ 1149.869631] Key type ._llcrypt unregistered [ 1158.314701] Key type ._llcrypt registered [ 1158.320123] Key type .llcrypt registered [ 1158.566318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1158.587291] alg: No test for adler32 (adler32-zlib) [ 1159.505659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1159.529778] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1166.865197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1178.889051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1178.944623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1178.991894] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1180.662326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_interrupt -r 1 [ 1180.702152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_interrupt -r 1 [ 1180.747763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_interrupt -r 1 [ 1180.802207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_interrupt -r 1 [ 1180.863334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_interrupt -r 1 [ 1180.913727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_interrupt -r 1 [ 1180.959840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_interrupt -r 1 [ 1181.022800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_interrupt -r 1 [ 1181.074921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1181.114092] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1181.129378] LNet: 1 local NIs in recovery (showing 1): 192.168.202.29@tcp [ 1181.408798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.430521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.455289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.481866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.508442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.536507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.570281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.607299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.623170] LNet: 60874:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1181.669375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.713437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.776609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.820241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.856044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.905943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.971787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1182.657892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_dropped -r 1 [ 1182.710910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_dropped -r 1 [ 1182.741713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_dropped -r 1 [ 1182.780973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_dropped -r 1 [ 1182.830269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_dropped -r 1 [ 1182.880550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_dropped -r 1 [ 1182.922606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_dropped -r 1 [ 1182.968383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_dropped -r 1 [ 1183.016431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1183.036859] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1183.041259] LNet: Skipped 2 previous similar messages [ 1183.275977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.314248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.340453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.363423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.389296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.432949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.470314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.503917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.532799] LNet: 60995:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.557252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.590599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.642289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.670591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.715266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.754955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.802941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1184.328129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_aborted -r 1 [ 1184.357568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_aborted -r 1 [ 1184.384726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_aborted -r 1 [ 1184.416326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_aborted -r 1 [ 1184.443571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_aborted -r 1 [ 1184.469244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_aborted -r 1 [ 1184.495547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_aborted -r 1 [ 1184.526236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_aborted -r 1 [ 1184.562945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1184.620568] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1184.624030] LNet: Skipped 2 previous similar messages [ 1184.857930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.887054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.965703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1185.018765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1185.078577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1185.138541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1185.168440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1185.194454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1185.210487] LNet: 61116:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1185.225863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1185.253987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1185.280694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1185.307554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1185.336354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1185.361768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1185.385156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1185.992035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_no_route -r 1 [ 1186.034403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_no_route -r 1 [ 1186.080068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_no_route -r 1 [ 1186.125560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_no_route -r 1 [ 1186.171416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_no_route -r 1 [ 1186.259571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_no_route -r 1 [ 1186.309081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_no_route -r 1 [ 1186.384823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_no_route -r 1 [ 1186.423537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1186.710744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.733747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1186.756370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.785699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.818439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.859219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.896890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.922194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.953847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.985991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1187.032664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.093970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1187.138722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1187.178967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1187.223696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1187.815732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_timeout -r 1 [ 1187.843871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_timeout -r 1 [ 1187.874693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_timeout -r 1 [ 1187.907705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_timeout -r 1 [ 1187.939474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_timeout -r 1 [ 1187.990343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_timeout -r 1 [ 1188.033042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_timeout -r 1 [ 1188.069705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_timeout -r 1 [ 1188.107868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1188.134385] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1188.143251] LNet: Skipped 5 previous similar messages [ 1188.376918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1188.414021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1188.445754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1188.466929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1188.486343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1188.506689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1188.536271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1188.564310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1188.582852] LNet: 61359:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1188.586951] LNet: 61359:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1188.600734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1188.625166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1188.661677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1188.686827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1188.712978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1188.740826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1188.769746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1189.258967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_error -r 1 [ 1189.291326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_error -r 1 [ 1189.323705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_error -r 1 [ 1189.363677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_error -r 1 [ 1189.408902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_error -r 1 [ 1189.448256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_error -r 1 [ 1189.495773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_error -r 1 [ 1189.543854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_error -r 1 [ 1189.583695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1189.915989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.952703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1190.001900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1190.039790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1190.071850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1190.102336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1190.132446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1190.157471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1190.196938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1190.229532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1190.274422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.308277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.344840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.366752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1190.415050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1197.287413] LNet: 61556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.293287] LNetError: 61556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1197.314559] LNet: Removed LNI 192.168.202.29@tcp [ 1198.394815] LNet: Removed LNI 192.168.202.29@tcp1 [ 1198.985652] Key type .llcrypt unregistered [ 1198.989584] Key type ._llcrypt unregistered [ 1219.349253] Key type ._llcrypt registered [ 1219.369376] Key type .llcrypt registered [ 1221.424664] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:17:35 (1788844655) [ 1222.518281] Key type .llcrypt unregistered [ 1222.521482] Key type ._llcrypt unregistered [ 1230.666763] Key type ._llcrypt registered [ 1230.668293] Key type .llcrypt registered [ 1230.933692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1230.961503] alg: No test for adler32 (adler32-zlib) [ 1231.926229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1231.962202] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1241.893876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1242.825478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e remote_dropped -r 1 [ 1242.865898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e remote_dropped -r 1 [ 1242.929711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1242.950580] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1243.355975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.387814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.431447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.476696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.531562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.561303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.603745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.639940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.667400] LNet: 62801:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.693466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.727744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.802945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.876834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.947365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.012500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1244.065668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1244.968988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e remote_error -r 1 [ 1245.056678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e remote_error -r 1 [ 1245.132615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1245.190220] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1245.534113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1245.586332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1245.627278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1245.657899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1245.697570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1245.743786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1245.769809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.799542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1245.813982] LNet: 62910:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.831973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1245.855598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.878468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.905235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.926503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.973642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1246.030028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1246.792660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e remote_timeout -r 1 [ 1246.840554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e remote_timeout -r 1 [ 1246.895536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1246.913805] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1247.209884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1247.251431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1247.280896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1247.316440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1247.344264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1247.375227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.409438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.440838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.456657] LNet: 63019:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.474429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.501585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.551098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.593604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.636991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1247.671936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1247.708355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1247.976824] LNet: 63060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.983460] LNetError: 63060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1248.003569] LNet: Removed LNI 192.168.202.29@tcp [ 1248.491530] Key type .llcrypt unregistered [ 1248.497166] Key type ._llcrypt unregistered [ 1269.686439] Key type ._llcrypt registered [ 1269.688228] Key type .llcrypt registered [ 1271.328400] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:18:25 (1788844705) [ 1272.319334] Key type .llcrypt unregistered [ 1272.322750] Key type ._llcrypt unregistered [ 1279.635889] Key type ._llcrypt registered [ 1279.637407] Key type .llcrypt registered [ 1279.872632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1279.891822] alg: No test for adler32 (adler32-zlib) [ 1280.816736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1280.843829] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1289.145664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1298.896756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1298.960397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1298.990330] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1301.049106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e remote_dropped -r 1 [ 1301.100308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e remote_dropped -r 1 [ 1301.147887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e remote_dropped -r 1 [ 1301.208883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e remote_dropped -r 1 [ 1301.251471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e remote_dropped -r 1 [ 1301.304241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e remote_dropped -r 1 [ 1301.341513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e remote_dropped -r 1 [ 1301.377812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e remote_dropped -r 1 [ 1301.415171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1301.435655] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1301.441679] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.129@tcp [ 1301.694320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1301.749620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1301.845598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1301.901229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1301.956826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1302.021823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1302.087712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1302.133282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1302.174991] LNet: 64526:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1302.193684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1302.221179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1302.267078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1302.315720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1302.343390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1302.388240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1302.432939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1302.493786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1303.088389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e remote_error -r 1 [ 1303.125982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e remote_error -r 1 [ 1303.161188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e remote_error -r 1 [ 1303.218979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e remote_error -r 1 [ 1303.253982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e remote_error -r 1 [ 1303.289446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e remote_error -r 1 [ 1303.318210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e remote_error -r 1 [ 1303.350883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e remote_error -r 1 [ 1303.388702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1303.406825] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1303.411829] LNet: Skipped 2 previous similar messages [ 1303.638117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1303.657725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1303.677122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1303.719889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1303.751307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1303.799640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1303.831762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1303.876495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1303.905735] LNet: 64649:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1303.923451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1303.961454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1303.998812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1304.019424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1304.045660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.075506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1304.120326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1304.145849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.605752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e remote_timeout -r 1 [ 1304.681908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e remote_timeout -r 1 [ 1304.744976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e remote_timeout -r 1 [ 1304.776613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e remote_timeout -r 1 [ 1304.820856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e remote_timeout -r 1 [ 1304.859149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e remote_timeout -r 1 [ 1304.919782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e remote_timeout -r 1 [ 1304.959789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e remote_timeout -r 1 [ 1304.991260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.129@tcp [ 1305.013052] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1305.243774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.289935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1305.329344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1305.359916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.390876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.421702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.451840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.485801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.500646] LNet: 64773:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.517961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.545878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.579700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.608753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1305.642296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.674822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1305.702807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1305.743863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1310.947756] LNet: 64851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.958147] LNetError: 64851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.964276] LNet: Removed LNI 192.168.202.29@tcp [ 1310.978335] LNet: Removed LNI 192.168.202.29@tcp1 [ 1311.504881] Key type .llcrypt unregistered [ 1311.507086] Key type ._llcrypt unregistered [ 1331.176742] Key type ._llcrypt registered [ 1331.178854] Key type .llcrypt registered [ 1332.600810] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:19:27 (1788844767) [ 1334.515921] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1336.970739] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:19:31 (1788844771) [ 1338.136507] Key type .llcrypt unregistered [ 1338.140661] Key type ._llcrypt unregistered [ 1345.654537] Key type ._llcrypt registered [ 1345.656670] Key type .llcrypt registered [ 1345.871543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1345.892915] alg: No test for adler32 (adler32-zlib) [ 1346.937512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1346.993968] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1357.471859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1358.383578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e network_timeout -r 1 [ 1358.419910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e network_timeout -r 1 [ 1358.452497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1358.471359] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1358.793774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1358.849689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1358.897177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1358.934430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1358.958269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1358.986185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1359.065683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1359.126956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1359.163314] LNet: 66390:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1359.195221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1359.258894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1359.306652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1359.329533] LNet: 1 local NIs in recovery (showing 1): 192.168.202.29@tcp [ 1359.332631] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1359.356299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1359.409161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1359.449529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1359.692264] LNet: 66429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.697179] LNetError: 66429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1359.720587] LNet: Removed LNI 192.168.202.29@tcp [ 1360.156694] Key type .llcrypt unregistered [ 1360.160916] Key type ._llcrypt unregistered [ 1382.961347] Key type ._llcrypt registered [ 1382.964760] Key type .llcrypt registered [ 1383.171836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1383.197725] alg: No test for adler32 (adler32-zlib) [ 1384.093984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1384.125466] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1392.206415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1402.949772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1403.005577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1403.050071] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1404.563022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e network_timeout -r 1 [ 1404.615860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e network_timeout -r 1 [ 1404.650168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e network_timeout -r 1 [ 1404.693538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e network_timeout -r 1 [ 1404.747903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e network_timeout -r 1 [ 1404.789517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e network_timeout -r 1 [ 1404.851276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e network_timeout -r 1 [ 1404.904104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e network_timeout -r 1 [ 1404.948899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1404.967400] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1405.207846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1405.239289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1405.276116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1405.314755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1405.353976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1405.393402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1405.418342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1405.440943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1405.461296] LNet: 67315:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1405.481175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1405.521907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1405.581398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1405.621861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1405.652931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1405.667569] LNet: 1 local NIs in recovery (showing 1): 192.168.202.29@tcp [ 1405.677283] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1405.679423] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.129@tcp [ 1405.689469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1405.694706] LNet: Skipped 1 previous similar message [ 1407.718817] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1409.761259] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1411.611982] LNet: 67389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.619807] LNetError: 67389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.631365] LNet: Removed LNI 192.168.202.29@tcp [ 1411.648491] LNet: Removed LNI 192.168.202.29@tcp1 [ 1412.208162] Key type .llcrypt unregistered [ 1412.217840] Key type ._llcrypt unregistered [ 1431.157625] Key type ._llcrypt registered [ 1431.159653] Key type .llcrypt registered [ 1432.485928] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:21:07 (1788844867) [ 1432.986027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1433.018156] alg: No test for adler32 (adler32-zlib) [ 1433.996476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.039309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1434.059595] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1434.589725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1434.604485] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1435.601777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 1435.708620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1435.754458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1435.784479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1435.809171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1435.834790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1435.859975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.29@tcp [ 1435.878439] LNetError: 68487:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.29@tcp to recovery [ 1436.064559] LNet: 1 local NIs in recovery (showing 1): 192.168.202.29@tcp [ 1436.065326] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1438.120359] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1438.320130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1442.209521] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1443.308313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1443.394890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1443.445328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1443.486326] LNet: 68594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1443.497493] LNetError: 68594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1443.510905] LNet: Removed LNI 192.168.202.29@tcp [ 1443.532057] LNet: Removed LNI 192.168.202.29@tcp1 [ 1443.557670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1443.650725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1443.689511] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1444.640788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1444.654663] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1445.400591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 1445.425654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1445.482568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1445.526962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1445.561679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1445.596641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1445.625881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1445.656588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.29@tcp [ 1445.676703] LNetError: 68867:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.29@tcp to recovery [ 1446.689887] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1449.248961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1452.832385] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1452.837128] LNet: Skipped 1 previous similar message [ 1457.339445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1457.450347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1457.514454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1466.594218] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:21:41 (1788844901) [ 1466.693385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1466.711886] LNet: 69578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.721688] LNetError: 69578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.726780] LNetError: 69578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1466.738326] LNet: Removed LNI 192.168.202.29@tcp [ 1466.757211] LNet: Removed LNI 192.168.202.29@tcp1 [ 1466.800672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1466.887648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1466.935626] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1468.139848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1468.200927] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1469.677775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 1469.730663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1469.768784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1469.805698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1469.839457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1469.885173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1469.920807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.29@tcp --health 0 [ 1470.043696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1470.944525] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.29@tcp [ 1470.945055] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1470.967840] LNet: Skipped 1 previous similar message [ 1484.980427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1485.168801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1485.215780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1485.252600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.29@tcp --health 500 [ 1485.511864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1492.543621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1492.664888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.705082] LNet: 70068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.713337] LNetError: 70068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.721553] LNetError: 70068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1492.737504] LNet: Removed LNI 192.168.202.29@tcp [ 1492.784590] LNet: Removed LNI 192.168.202.29@tcp1 [ 1492.865636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.961279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1493.006027] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1494.248434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1494.265834] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1495.465853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 1495.520721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1495.619156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1495.654216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1495.682650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1495.718960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1495.756343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1495.790829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.29@tcp --health 0 [ 1496.995382] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1497.002934] LNet: Skipped 3 previous similar messages [ 1503.723315] Lustre: DEBUG MARKER: sanity-lnet test_211: @@@@@@ FAIL: Unexpected ping count [ 1516.546071] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:22:30 (1788844950) [ 1517.315276] LNet: 71156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.335322] LNetError: 71156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.343461] LNetError: 71156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1517.362635] LNet: Removed LNI 192.168.202.29@tcp [ 1517.399308] LNet: Removed LNI 192.168.202.29@tcp1 [ 1518.613437] Key type .llcrypt unregistered [ 1518.617046] Key type ._llcrypt unregistered [ 1526.904083] Key type ._llcrypt registered [ 1526.906209] Key type .llcrypt registered [ 1527.285293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1527.316272] alg: No test for adler32 (adler32-zlib) [ 1528.306497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1528.349224] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1539.159931] Lustre: DEBUG MARKER: Initial discovery [ 1539.618271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.129@tcp [ 1542.270814] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1542.532191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1542.578330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.129@tcp [ 1542.608759] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1542.666979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1542.688275] LNet: 1 local NIs in recovery (showing 1): 192.168.202.29@tcp [ 1542.710972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.129@tcp -r 1 -m GET -l 3 [ 1542.761036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.129@tcp -r 1 -m GET -e local_error [ 1542.792264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.129@tcp -r 1 -m PUT -l 6 [ 1544.302160] Lustre: DEBUG MARKER: Force oleg229-client.virtnet to discover 192.168.202.129@tcp (in background) [ 1544.686913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.129@tcp [ 1546.329103] Lustre: DEBUG MARKER: Force oleg229-server to discover 192.168.202.29@tcp [ 1547.813140] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1549.125441] Lustre: DEBUG MARKER: Wait for 71917 [ 1551.012194] Lustre: DEBUG MARKER: Finished wait on 71917 [ 1551.222105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1551.271811] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1551.277737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1551.583543] LNet: 72367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1551.589356] LNetError: 72367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1551.607559] LNet: Removed LNI 192.168.202.29@tcp [ 1551.991818] Key type .llcrypt unregistered [ 1551.994854] Key type ._llcrypt unregistered [ 1573.525688] Key type ._llcrypt registered [ 1573.531696] Key type .llcrypt registered [ 1574.964816] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:23:29 (1788845009) [ 1575.724396] Key type .llcrypt unregistered [ 1575.734513] Key type ._llcrypt unregistered [ 1582.251448] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1582.509802] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1582.517317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1582.521221] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1585.170479] Key type ._llcrypt registered [ 1585.172390] Key type .llcrypt registered [ 1585.402641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1585.415707] alg: No test for adler32 (adler32-zlib) [ 1586.349420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.475842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1586.499931] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1587.327502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1587.357596] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1595.952539] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:23:50 (1788845030) [ 1596.313167] LNet: 74436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.322799] LNetError: 74436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1596.329731] LNet: Removed LNI 10.1.2.3@tcp [ 1596.360710] LNet: Removed LNI 192.168.202.29@tcp [ 1597.053319] Key type .llcrypt unregistered [ 1597.058507] Key type ._llcrypt unregistered [ 1604.757288] Key type ._llcrypt registered [ 1604.759986] Key type .llcrypt registered [ 1604.928714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1604.940603] alg: No test for adler32 (adler32-zlib) [ 1605.934607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.003295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1606.027699] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1607.373284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1607.411508] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1608.817552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.29@tcp [ 1608.897591] LNetError: Unexpected error -22 connecting to 192.168.202.29@tcp at host 192.168.202.29:988 [ 1616.845261] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:24:11 (1788845051) [ 1616.986595] LNet: 75559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.990572] LNetError: 75559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1616.996562] LNet: Removed LNI 10.1.2.3@tcp [ 1617.015766] LNet: Removed LNI 192.168.202.29@tcp [ 1617.893051] Key type .llcrypt unregistered [ 1617.900918] Key type ._llcrypt unregistered [ 1624.738723] Key type ._llcrypt registered [ 1624.744530] Key type .llcrypt registered [ 1625.090531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1625.111249] alg: No test for adler32 (adler32-zlib) [ 1626.317629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.464873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1626.514956] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1627.837403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1627.913227] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 1629.348184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.29@tcp1 --nid 192.168.202.29@tcp2 [ 1637.373735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.29@tcp1 192.168.202.29@tcp2 [ 1637.423707] LNetError: 76399:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.29@tcp1 to 192.168.202.29@tcp2 - Route table may be misconfigured [ 1637.474906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.29@tcp2 192.168.202.29@tcp1 [ 1647.219043] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:24:41 (1788845081) [ 1648.083683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.108674] LNet: 76973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.119892] LNetError: 76973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1648.131795] LNet: Removed LNI 192.168.202.29@tcp1 [ 1649.190904] LNet: Removed LNI 192.168.202.29@tcp2 [ 1649.300328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.484719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1649.567426] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1651.262486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1651.288601] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1652.520842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 1652.608304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.29@tcp 192.168.202.29@tcp [ 1652.689568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.29@tcp1 192.168.202.29@tcp1 [ 1652.756027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.29@tcp -d 192.168.202.29@tcp -e network_timeout [ 1652.809897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -e network_timeout [ 1652.865495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -e network_timeout [ 1652.911263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -e network_timeout [ 1652.965766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 1653.005214] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1653.054295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1653.270229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1653.481796] LNet: 1 local NIs in recovery (showing 1): 192.168.202.29@tcp [ 1653.482722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1661.739093] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:24:56 (1788845096) [ 1661.804675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1661.830899] LNet: 77833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.836648] LNetError: 77833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1661.843043] LNetError: 77833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1661.848799] LNet: Removed LNI 192.168.202.29@tcp [ 1662.901399] LNet: Removed LNI 192.168.202.29@tcp1 [ 1662.940802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.130148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1663.421861] LNet: 77873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.992553] Key type .llcrypt unregistered [ 1663.995917] Key type ._llcrypt unregistered [ 1679.247611] Key type ._llcrypt registered [ 1679.249312] Key type .llcrypt registered [ 1680.911442] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:25:15 (1788845115) [ 1681.256735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1681.274710] alg: No test for adler32 (adler32-zlib) [ 1682.252974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.987988] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1686.276836] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:25:20 (1788845120) [ 1686.630220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.657049] LNet: 78969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.683575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.776929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1686.817070] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1688.017089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1688.065901] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1689.257512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 1689.295357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp1 [ 1689.335544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp1 [ 1697.444847] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:25:31 (1788845131) [ 1699.624774] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg229-server [ 1701.351832] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:25:36 (1788845136) [ 1703.420802] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg229-server [ 1705.619413] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:25:40 (1788845140) [ 1708.400411] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg229-server [ 1710.540406] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:25:45 (1788845145) [ 1713.099262] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg229-server [ 1714.967244] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:25:49 (1788845149) [ 1716.863542] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg229-server [ 1718.791719] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:25:53 (1788845153) [ 1721.109682] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg229-server [ 1722.901403] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:25:57 (1788845157) [ 1724.471916] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg229-server [ 1726.692937] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:26:01 (1788845161) [ 1728.548548] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg229-server [ 1730.560129] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:26:05 (1788845165) [ 1732.479848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg229-server [ 1735.262940] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:26:09 (1788845169) [ 1735.480014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.523949] LNet: 82843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.537338] LNetError: 82843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.608911] LNet: Removed LNI 192.168.202.29@tcp [ 1736.632146] LNetError: 82843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.660304] LNet: Removed LNI 192.168.202.29@tcp1 [ 1736.719713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.869260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1736.905227] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1738.010773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1738.108428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 1738.260469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.313815] LNet: 82995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.332223] LNetError: 82995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1738.357579] LNet: Removed LNI 192.168.202.29@tcp [ 1738.398389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.504690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1738.532037] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1739.714635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 1739.990740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.032093] LNet: Removed LNI 192.168.202.29@tcp [ 1741.068105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.133784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1741.178480] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1742.041957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1742.166465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 1742.457674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.492720] LNet: 83287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.497125] LNet: 83287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1742.514026] LNetError: 83287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.522979] LNetError: 83287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1742.540260] LNet: Removed LNI 192.168.202.29@tcp [ 1742.584803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.676489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1742.713352] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1743.587156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 1743.871545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.936262] LNet: Removed LNI 192.168.202.29@tcp [ 1744.992514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.084131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1745.125502] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1745.769350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1745.878496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 1746.150941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1747.245744] LNet: Removed LNI 192.168.202.29@tcp [ 1747.275983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.316595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1747.344429] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1748.007348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 1748.190963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.218749] LNet: 83725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.222784] LNet: 83725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1748.230642] LNetError: 83725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1748.239649] LNetError: 83725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1749.293217] LNet: Removed LNI 192.168.202.29@tcp [ 1749.342443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.418854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1749.445257] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1750.497892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1750.663643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 1750.956853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.040755] LNet: Removed LNI 192.168.202.29@tcp [ 1752.086728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.148603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1752.176350] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1753.206929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 1753.522355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1754.613096] LNet: Removed LNI 192.168.202.29@tcp [ 1754.662199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.784886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1754.824613] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1755.516992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1755.590946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 1755.762700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1756.839390] LNet: Removed LNI 192.168.202.29@tcp [ 1756.862748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.917435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1756.948126] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1757.700438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 1757.911807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.933830] LNet: 84311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.946123] LNet: 84311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1757.957561] LNetError: 84311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.968739] LNetError: 84311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1757.986328] LNet: Removed LNI 192.168.202.29@tcp [ 1758.012551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.073626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1758.117524] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1758.878854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1758.950730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 1759.208091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.300124] LNet: Removed LNI 192.168.202.29@tcp [ 1760.334641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.408758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1760.476163] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1761.478703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 1761.741813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.792560] LNet: Removed LNI 192.168.202.29@tcp [ 1762.820703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.894402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1762.940551] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1763.717788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1763.810389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.841957] LNet: Removed LNI 192.168.202.29@tcp [ 1763.868313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.926375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1763.950095] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1764.979238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1765.069841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.107523] LNet: Removed LNI 192.168.202.29@tcp [ 1765.136563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.202669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1765.316832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.375417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.435710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1765.601198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.653171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.756858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1765.827734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.859619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.933983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1765.981764] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1767.035846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.071302] LNet: Removed LNI 192.168.202.29@tcp [ 1767.109671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.219813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1774.370765] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:26:49 (1788845209) [ 1774.643956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.671209] LNet: 85624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.689063] LNet: 85624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1774.736550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.817393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1774.865165] LNet: Added LNI 192.168.202.29@tcp231 [8/256/0/180] [ 1776.210833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.236918] LNetError: 85762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.241476] LNetError: 85762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1776.256735] LNet: Removed LNI 192.168.202.29@tcp231 [ 1776.283878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1776.378610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1776.437357] LNet: Added LNI 192.168.202.29@tcp231 [8/256/0/180] [ 1776.446089] LNet: 85770:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1776.915038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1776.975478] LNet: Removed LNI 192.168.202.29@tcp231 [ 1777.010446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1777.037665] LNet: Added LNI 192.168.202.29@tcp231 [8/256/0/180] [ 1778.396089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.444593] LNet: Removed LNI 192.168.202.29@tcp231 [ 1778.497307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.609887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1778.643662] LNet: Added LNI 192.168.202.29@tcp231 [8/256/0/0] [ 1778.649539] LNet: 85923:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1779.173342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1779.206299] LNet: Removed LNI 192.168.202.29@tcp231 [ 1779.217148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1779.255410] LNet: Added LNI 192.168.202.29@tcp231 [8/256/0/0] [ 1780.191621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1780.227676] LNet: Removed LNI 192.168.202.29@tcp231 [ 1780.242453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.290776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1780.311055] LNet: Added LNI 192.168.202.29@tcp231 [8/256/0/60] [ 1780.314793] LNet: 86074:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1780.687956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1780.714948] LNet: Removed LNI 192.168.202.29@tcp231 [ 1780.726970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1780.757292] LNet: Added LNI 192.168.202.29@tcp231 [8/256/0/60] [ 1781.863577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.921335] LNet: Removed LNI 192.168.202.29@tcp231 [ 1781.941344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.028415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1782.068250] LNet: Added LNI 192.168.202.29@tcp231 [8/256/0/180] [ 1782.614412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1782.651336] LNet: Removed LNI 192.168.202.29@tcp231 [ 1782.674794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1782.715738] LNet: Added LNI 192.168.202.29@tcp231 [8/256/0/180] [ 1790.292367] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:27:05 (1788845225) [ 1790.946614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1790.998658] LNet: Removed LNI 192.168.202.29@tcp231 [ 1791.040486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.134736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1791.208996] LNet: Added LNI 192.168.202.29@tcp232 [8/256/0/180] [ 1792.101129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1792.161708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1792.208298] LNet: Removed LNI 192.168.202.29@tcp232 [ 1792.249281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1792.301323] LNet: Added LNI 192.168.202.29@tcp232 [8/256/0/180] [ 1793.180961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1793.207965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1793.236953] LNet: Removed LNI 192.168.202.29@tcp232 [ 1793.253229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1793.275813] LNet: Added LNI 192.168.202.29@tcp232 [8/256/0/180] [ 1794.050807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1794.077734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1794.101860] LNet: Removed LNI 192.168.202.29@tcp232 [ 1794.116914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1794.137458] LNet: Added LNI 192.168.202.29@tcp232 [8/256/0/180] [ 1794.866612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1794.897273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1794.923832] LNet: Removed LNI 192.168.202.29@tcp232 [ 1794.939708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1802.202721] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:27:16 (1788845236) [ 1803.292498] Key type .llcrypt unregistered [ 1803.307503] Key type ._llcrypt unregistered [ 1812.089352] Key type ._llcrypt registered [ 1812.095657] Key type .llcrypt registered [ 1812.437570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1812.456395] alg: No test for adler32 (adler32-zlib) [ 1813.533693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1813.617426] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1825.334625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1837.357936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1837.382714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1837.409777] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1838.870577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_interrupt -r 2 [ 1838.925726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_interrupt -r 2 [ 1838.991684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_interrupt -r 2 [ 1839.026396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_interrupt -r 2 [ 1839.077174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_interrupt -r 2 [ 1839.108754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_interrupt -r 2 [ 1839.137225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_interrupt -r 2 [ 1839.157747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_interrupt -r 2 [ 1839.222687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.29@tcp 192.168.202.129@tcp [ 1839.260980] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1839.266437] LNet: 1 local NIs in recovery (showing 1): 192.168.202.29@tcp [ 1839.587730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1839.624236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1839.660794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1839.716584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1839.851620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1839.919433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1839.994965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1840.065848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1840.109575] LNet: 88723:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1840.146124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1840.182163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1840.228433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1840.249725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1840.272419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1840.303961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1841.184508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_dropped -r 2 [ 1841.231111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_dropped -r 2 [ 1841.270688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_dropped -r 2 [ 1841.311116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_dropped -r 2 [ 1841.350557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_dropped -r 2 [ 1841.382764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_dropped -r 2 [ 1841.411983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_dropped -r 2 [ 1841.446145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_dropped -r 2 [ 1841.475141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.29@tcp 192.168.202.129@tcp [ 1841.497401] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1841.828801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1841.859991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1841.901977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1841.950626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1842.002764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1842.056766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1842.082674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1842.146256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1842.179028] LNet: 88843:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1842.199433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1842.238723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1842.277401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1842.344252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1842.409571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1842.489661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1843.219980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_aborted -r 2 [ 1843.317716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_aborted -r 2 [ 1843.387451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_aborted -r 2 [ 1843.460572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_aborted -r 2 [ 1843.531357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_aborted -r 2 [ 1843.626055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_aborted -r 2 [ 1843.703848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_aborted -r 2 [ 1843.782966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_aborted -r 2 [ 1843.854604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.29@tcp 192.168.202.129@tcp [ 1843.887813] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1844.214165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1844.265660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1844.307163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1844.342203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1844.364302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1844.403778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1844.464801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1844.514467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1844.536563] LNet: 88964:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1844.564321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1844.650778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1844.709551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1844.752941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1844.799871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1844.846260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1845.580910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_no_route -r 2 [ 1845.625880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_no_route -r 2 [ 1845.668514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_no_route -r 2 [ 1845.718129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_no_route -r 2 [ 1845.756319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_no_route -r 2 [ 1845.832528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_no_route -r 2 [ 1845.920344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_no_route -r 2 [ 1846.040535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_no_route -r 2 [ 1846.143618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.29@tcp 192.168.202.129@tcp [ 1846.172678] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1846.355214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1846.372169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1846.392707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1846.418916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1846.443768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1846.469498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1846.493905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1846.518362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1846.541831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1846.567070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1846.592898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1846.673370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1846.741834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1846.789725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1847.885438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e local_timeout -r 2 [ 1847.976780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e local_timeout -r 2 [ 1848.095417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e local_timeout -r 2 [ 1848.174759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e local_timeout -r 2 [ 1848.259394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e local_timeout -r 2 [ 1848.301119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e local_timeout -r 2 [ 1848.385389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e local_timeout -r 2 [ 1848.442446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e local_timeout -r 2 [ 1848.528337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.29@tcp 192.168.202.129@tcp [ 1848.866468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1848.924401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1848.997809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1849.040384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1849.109449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1849.156284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1849.193282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1849.231457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1849.273843] LNet: 89205:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1849.284634] LNet: 89205:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1849.337405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1849.392213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1849.424209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1849.463420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1849.523753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1849.588783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1850.159930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp -m GET -e remote_dropped -r 2 [ 1850.192282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.129@tcp1 -m GET -e remote_dropped -r 2 [ 1850.225440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp -m GET -e remote_dropped -r 2 [ 1850.263707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp -d 192.168.202.29@tcp1 -m GET -e remote_dropped -r 2 [ 1850.334979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp -m GET -e remote_dropped -r 2 [ 1850.385499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.29@tcp1 -d 192.168.202.129@tcp1 -m GET -e remote_dropped -r 2 [ 1850.428518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp -m GET -e remote_dropped -r 2 [ 1850.483784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.29@tcp1 -d 192.168.202.29@tcp1 -m GET -e remote_dropped -r 2 [ 1850.527745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.29@tcp 192.168.202.129@tcp [ 1850.545513] LNet: There was an unexpected network error while writing to 192.168.202.129: rc = -22 [ 1850.549854] LNet: Skipped 1 previous similar message [ 1850.552221] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.129@tcp [ 1850.872423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1850.918309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1850.989685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1851.040188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1851.099278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1851.156712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1851.220752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1851.264257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1851.313351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1851.342407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1851.370808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1851.404825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1851.432880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1851.465386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1859.941849] LNet: 89399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.947565] LNetError: 89399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1859.958427] LNet: Removed LNI 192.168.202.29@tcp [ 1859.972594] LNet: Removed LNI 192.168.202.29@tcp1 [ 1860.440672] Key type .llcrypt unregistered [ 1860.444270] Key type ._llcrypt unregistered [ 1881.364677] Key type ._llcrypt registered [ 1881.367701] Key type .llcrypt registered [ 1883.410348] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 01:28:37 (1788845317) [ 1883.983452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1884.010887] alg: No test for adler32 (adler32-zlib) [ 1885.024608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1885.093973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1885.125329] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1886.155191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1886.195752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.29@tcp [ 1886.226566] LNetError: 90377:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.29@tcp to recovery [ 1886.348448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.29@tcp [ 1886.508948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1894.132488] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 01:28:48 (1788845328) [ 1894.332409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1894.355541] LNet: 90966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1894.362930] LNetError: 90966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.381490] LNet: Removed LNI 192.168.202.29@tcp [ 1894.406883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.457859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1894.477206] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1895.293588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1895.312658] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 1896.184493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.29@tcp [ 1896.201810] LNetError: 91233:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.29@tcp to recovery [ 1896.480925] LNet: 1 local NIs in recovery (showing 1): 192.168.202.29@tcp [ 1896.482426] LNet: There was an unexpected network error while writing to 192.168.202.29: rc = -22 [ 1897.459894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.29@tcp [ 1906.418468] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 01:29:00 (1788845340) [ 1908.783525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg229-server [ 1911.014905] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 01:29:05 (1788845345) [ 1913.114842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg229-server [ 1914.991446] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 01:29:09 (1788845349) [ 1917.075762] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg229-server [ 1918.791957] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:29:13 (1788845353) [ 1918.881487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1918.941362] LNet: 92849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1918.953538] LNetError: 92849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.968956] LNet: Removed LNI 192.168.202.29@tcp [ 1919.972446] LNetError: 92849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.013743] LNet: Removed LNI 192.168.202.29@tcp1 [ 1920.069385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.188386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1920.233629] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1921.140628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1921.277487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1921.357723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1921.547994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1921.658065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1921.795745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1928.562773] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:29:23 (1788845363) [ 1928.669443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1928.711094] LNet: 93584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.717857] LNetError: 93584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.742272] LNet: Removed LNI 192.168.202.29@tcp [ 1928.783092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.868472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1928.899257] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1936.459858] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:29:31 (1788845371) [ 1936.966830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1936.997275] LNet: 94292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.010028] LNetError: 94292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1937.041416] LNet: Removed LNI 192.168.202.29@tcp [ 1937.082576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1937.153483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1937.177172] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1944.777132] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:29:39 (1788845379) [ 1946.562935] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1948.741771] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:29:43 (1788845383) [ 1949.559570] LNet: 95327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1949.564452] LNetError: 95327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1949.577967] LNet: Removed LNI 192.168.202.29@tcp [ 1950.170648] Key type .llcrypt unregistered [ 1950.172215] Key type ._llcrypt unregistered [ 1959.784451] Key type ._llcrypt registered [ 1959.791116] Key type .llcrypt registered [ 1960.157895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1960.194481] alg: No test for adler32 (adler32-zlib) [ 1961.213900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1961.258590] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 1971.225720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 1977.681586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.129@tcp [ 1978.073713] LNet: 95742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1978.079296] LNetError: 95742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1978.092062] LNet: Removed LNI 192.168.202.29@tcp [ 1978.503744] Key type .llcrypt unregistered [ 1978.505346] Key type ._llcrypt unregistered [ 1990.029457] Key type ._llcrypt registered [ 1990.035052] Key type .llcrypt registered [ 1991.242775] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1993.137759] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1995.212280] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:30:29 (1788845429) [ 1996.942909] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1998.872501] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:30:33 (1788845433) [ 2000.824369] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg229-server [ 2002.747279] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:30:37 (1788845437) [ 2004.617877] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg229-server [ 2006.894180] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:30:41 (1788845441) [ 2008.102428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2008.250739] alg: No test for adler32 (adler32-zlib) [ 2009.341622] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2009.661548] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2011.417931] Key type lgssc registered [ 2012.864835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2037.474606] Key type lgssc unregistered [ 2037.930403] LNet: 98912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.942859] LNetError: 98912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2037.969436] LNet: Removed LNI 192.168.202.29@tcp [ 2039.029622] Key type .llcrypt unregistered [ 2039.037732] Key type ._llcrypt unregistered [ 2054.959244] Key type ._llcrypt registered [ 2054.961426] Key type .llcrypt registered [ 2056.085561] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 01:31:31 (1788845491) [ 2056.662184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2056.676789] alg: No test for adler32 (adler32-zlib) [ 2057.641958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2057.694589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2057.733628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2057.778780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2057.827254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2064.128395] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 01:31:38 (1788845498) [ 2064.293995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2064.336348] LNet: 100323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2064.384752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2064.522874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2064.557129] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2065.772950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2065.794987] LNet: 100468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2065.802257] LNetError: 100468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2065.816217] LNet: Removed LNI 192.168.202.29@tcp [ 2065.849407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2065.919509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2065.941251] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2066.804877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2066.836419] LNet: 100496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.845860] LNetError: 100496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2066.869440] LNet: Removed LNI 192.168.202.29@tcp [ 2066.918138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2066.963397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2066.985725] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2074.596458] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 01:31:49 (1788845509) [ 2074.648618] LNet: 101077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2074.653944] LNetError: 101077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2074.675656] LNet: Removed LNI 192.168.202.29@tcp [ 2075.349602] Key type .llcrypt unregistered [ 2075.354095] Key type ._llcrypt unregistered [ 2081.793730] Key type ._llcrypt registered [ 2081.797390] Key type .llcrypt registered [ 2082.083043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2082.097551] alg: No test for adler32 (adler32-zlib) [ 2083.050444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2083.124232] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2083.127255] LNet: *** cfs_fail_loc=e003, val=100*** [ 2084.137477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 2084.155956] LNet: *** cfs_fail_loc=e003, val=100*** [ 2084.158770] LNet: Skipped 2 previous similar messages [ 2084.165507] Ping reply received with mapping: [ 2084.168498] found metadata reply NI[192.168.202.29@tcp] [ 2084.170738] 1d ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2084.174313] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2084.177438] 5a 5a 5a 5a 5a 5a 5a 5a [ 2084.362883] LNet: 101481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2084.369182] LNetError: 101481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2084.385985] LNet: Removed LNI 192.168.202.29@tcp [ 2085.123661] Key type .llcrypt unregistered [ 2085.125472] Key type ._llcrypt unregistered [ 2098.212163] Key type ._llcrypt registered [ 2098.214618] Key type .llcrypt registered [ 2099.852300] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 01:32:14 (1788845534) [ 2100.498405] Key type .llcrypt unregistered [ 2100.502716] Key type ._llcrypt unregistered [ 2108.235092] Key type ._llcrypt registered [ 2108.237679] Key type .llcrypt registered [ 2108.501944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2108.536758] alg: No test for adler32 (adler32-zlib) [ 2109.903366] LNetError: 102518:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2110.410529] Key type .llcrypt unregistered [ 2110.415093] Key type ._llcrypt unregistered [ 2116.695732] Key type ._llcrypt registered [ 2116.700776] Key type .llcrypt registered [ 2118.320649] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 01:32:32 (1788845552) [ 2119.096441] Key type .llcrypt unregistered [ 2119.099276] Key type ._llcrypt unregistered [ 2127.820891] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2127.832148] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2130.596153] Key type ._llcrypt registered [ 2130.603964] Key type .llcrypt registered [ 2130.785721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2130.796398] alg: No test for adler32 (adler32-zlib) [ 2131.818492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2131.954295] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2144.091121] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 01:32:58 (1788845578) [ 2144.398794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.452102] LNet: 104289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.471728] LNetError: 104289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2144.516667] LNet: Removed LNI 10.1.2.3@tcp [ 2144.638353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.923336] Key type .llcrypt unregistered [ 2145.924855] Key type ._llcrypt unregistered [ 2155.742931] Key type ._llcrypt registered [ 2155.744792] Key type .llcrypt registered [ 2155.930920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2155.942163] alg: No test for adler32 (adler32-zlib) [ 2156.849549] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2156.947119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 2157.159586] LNet: 104578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2157.176238] LNetError: 104578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2157.207174] LNet: Removed LNI 192.168.202.29@tcp [ 2157.766462] Key type .llcrypt unregistered [ 2157.776586] Key type ._llcrypt unregistered [ 2171.224268] Key type ._llcrypt registered [ 2171.229955] Key type .llcrypt registered [ 2172.733886] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 01:33:27 (1788845607) [ 2173.288391] Key type .llcrypt unregistered [ 2173.290381] Key type ._llcrypt unregistered [ 2179.809447] Key type ._llcrypt registered [ 2179.811692] Key type .llcrypt registered [ 2180.048956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2180.070611] alg: No test for adler32 (adler32-zlib) [ 2189.274767] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 01:33:43 (1788845623) [ 2189.397680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.457825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.566748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2189.598767] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2190.432951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2190.481692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2190.522107] LNet: Removed LNI 192.168.202.29@tcp [ 2190.921653] LNet: 106317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.490640] Key type .llcrypt unregistered [ 2191.495351] Key type ._llcrypt unregistered [ 2203.699216] Key type ._llcrypt registered [ 2203.701641] Key type .llcrypt registered [ 2205.099832] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 01:33:59 (1788845639) [ 2205.477300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2205.491700] alg: No test for adler32 (adler32-zlib) [ 2206.384897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2206.443305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2206.462185] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2207.275590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 2207.301409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.29@tcp [ 2207.318564] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2207.321751] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2207.324791] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2207, 0, 0 [ 2207.331414] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2207.335616] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2207.340698] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2207.343802] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2207, 0, 0 [ 2207.349514] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2207.352420] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2207.356101] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2207.360779] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2207, 0, 2257 [ 2207.367006] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2207.370525] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2207.374881] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2207.379980] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2207, 0, 2257 [ 2207.385929] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2213.476401] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 01:34:08 (1788845648) [ 2213.822315] LNet: 107853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2213.829916] LNetError: 107853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.891057] LNet: Removed LNI 192.168.202.29@tcp [ 2215.516513] Key type .llcrypt unregistered [ 2215.519016] Key type ._llcrypt unregistered [ 2222.611208] Key type ._llcrypt registered [ 2222.613794] Key type .llcrypt registered [ 2222.841497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2222.856899] alg: No test for adler32 (adler32-zlib) [ 2223.778268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2223.809814] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2232.593233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 2243.234464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.324556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2243.367975] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 2244.287158] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2244.423758] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2244.437405] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2244.439422] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2246.854943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2246.898692] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2258.003345] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 01:34:53 (1788845693) [ 2258.303723] LNet: 109361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2258.315831] LNetError: 109361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.324473] LNet: Removed LNI 192.168.202.29@tcp [ 2259.360477] LNet: Removed LNI 192.168.202.29@tcp1 [ 2259.364174] LNetError: 109361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2259.382187] LNetError: 109361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2259.401670] LNet: Removed LNI 10.1.2.3@tcp99 [ 2260.172887] Key type .llcrypt unregistered [ 2260.174575] Key type ._llcrypt unregistered [ 2266.731604] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2266.738832] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2269.216312] Key type ._llcrypt registered [ 2269.218548] Key type .llcrypt registered [ 2269.375928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2269.390526] alg: No test for adler32 (adler32-zlib) [ 2270.291067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2270.342259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2270.370684] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2271.136521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2271.158863] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2273.084527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2273.135449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2273.177296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2273.196350] LNetError: 110103:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2273.237919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2273.293748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2273.700077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2273.796959] LNet: 110123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.813783] LNetError: 110123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2273.824253] LNet: Removed LNI 10.1.2.3@tcp [ 2274.861622] LNet: Removed LNI 192.168.202.29@tcp [ 2275.560633] Key type .llcrypt unregistered [ 2275.566532] Key type ._llcrypt unregistered [ 2288.654954] Key type ._llcrypt registered [ 2288.657859] Key type .llcrypt registered [ 2290.725902] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 01:35:24 (1788845724) [ 2291.574642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2291.603337] alg: No test for adler32 (adler32-zlib) [ 2292.685587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2292.803691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2292.852965] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2294.021360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 2294.164627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg229-client@tcp [ 2302.559791] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 01:35:37 (1788845737) [ 2303.135584] LNet: 111653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2303.140732] LNetError: 111653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2303.152536] LNet: Removed LNI 192.168.202.29@tcp [ 2303.852567] Key type .llcrypt unregistered [ 2303.855363] Key type ._llcrypt unregistered [ 2312.020536] Key type ._llcrypt registered [ 2312.027558] Key type .llcrypt registered [ 2312.246480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2312.297637] alg: No test for adler32 (adler32-zlib) [ 2313.344443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2313.378195] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2313.421914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2321.405330] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 01:35:55 (1788845755) [ 2323.444838] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2325.244430] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 01:35:59 (1788845759) [ 2327.068148] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2328.990757] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 01:36:03 (1788845763) [ 2329.190792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2329.235989] LNet: 113088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2329.241926] LNetError: 113088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2329.254681] LNet: Removed LNI 192.168.202.29@tcp [ 2329.265933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2329.337786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2329.391816] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2330.200065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2330.273860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2337.769554] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 01:36:12 (1788845772) [ 2338.192862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2338.220901] LNet: 113797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2338.231954] LNetError: 113797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2338.248255] LNet: Removed LNI 192.168.202.29@tcp [ 2338.292963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2338.346675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2338.370742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2339.155319] Key type .llcrypt unregistered [ 2339.157371] Key type ._llcrypt unregistered [ 2352.056931] Key type ._llcrypt registered [ 2352.059572] Key type .llcrypt registered [ 2353.247851] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 01:36:28 (1788845788) [ 2353.670973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2353.686737] alg: No test for adler32 (adler32-zlib) [ 2354.618821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2354.685859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2354.714174] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2355.383733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2355.431766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2363.345837] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 01:36:37 (1788845797) [ 2363.474373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2363.522674] LNet: 115346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2363.533451] LNetError: 115346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2363.572187] LNet: Removed LNI 192.168.202.29@tcp [ 2363.598818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2363.667455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2363.698544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2371.100758] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 01:36:45 (1788845805) [ 2371.254204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2371.294611] LNet: 115928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2371.339185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.443841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2371.973653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2372.016409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2372.085829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2372.428842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2372.458928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2372.530328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2372.547641] LNet: 115965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2372.575965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2372.640331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2373.073983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2373.107216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2373.195925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2373.600828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2373.646376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2379.938555] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 01:36:54 (1788845814) [ 2380.212106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2380.246675] LNet: 116565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2380.272583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.336470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2380.355263] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2380.896345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2380.912956] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 2381.582302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2381.614480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2381.814802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2381.855630] LNetError: 116844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2382.881249] LNet: Removed LNI 192.168.202.29@tcp [ 2382.884965] LNetError: 116844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2382.905557] LNet: Removed LNI 192.168.202.29@tcp2 [ 2382.973836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2383.030490] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2383.034469] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 2383.038348] LNet: 116850:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2383.338598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2383.468867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2384.544466] LNet: Removed LNI 192.168.202.29@tcp [ 2384.556791] LNet: Removed LNI 192.168.202.29@tcp2 [ 2384.587884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2384.644220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2384.673353] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2385.342609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2385.360584] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 2386.134893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2386.187259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2386.227253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2386.436471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2386.458299] LNet: 117172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2386.464991] LNet: 117172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2386.473328] LNetError: 117172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2386.478947] LNetError: 117172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2386.485541] LNet: Removed LNI 192.168.202.29@tcp [ 2387.558804] LNet: Removed LNI 192.168.202.29@tcp2 [ 2387.586535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2387.685413] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2387.697993] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 2387.707383] LNet: 117178:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2388.478361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2388.703540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2388.724819] LNet: Removed LNI 192.168.202.29@tcp [ 2389.810319] LNet: Removed LNI 192.168.202.29@tcp2 [ 2389.871770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2389.960374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2390.003580] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2390.769316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2390.786095] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 2391.774590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2391.874900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2392.160437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.204196] LNetError: 117495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2392.210163] LNetError: 117495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2392.219540] LNet: Removed LNI 192.168.202.29@tcp [ 2392.234278] LNet: Removed LNI 192.168.202.29@tcp2 [ 2392.268281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2392.378210] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2392.388575] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 2392.398605] LNet: 117501:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2393.082419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2393.477810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2394.528221] LNet: Removed LNI 192.168.202.29@tcp [ 2394.549991] LNet: Removed LNI 192.168.202.29@tcp2 [ 2394.596310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2394.659344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2394.698667] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2395.644250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2395.668648] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 2396.791703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2396.852706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2396.935704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2397.309750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2397.328807] LNet: 117822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2397.343296] LNet: 117822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2398.368198] LNet: Removed LNI 192.168.202.29@tcp [ 2399.401062] LNet: Removed LNI 192.168.202.29@tcp2 [ 2399.444858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2399.533495] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2399.539906] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 2399.544184] LNet: 117829:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2400.269188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2400.540637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2400.580879] LNetError: 117866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.589615] LNetError: 117866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2400.594906] LNet: Removed LNI 192.168.202.29@tcp [ 2401.640954] LNet: Removed LNI 192.168.202.29@tcp2 [ 2401.703034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2401.798425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2401.843061] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2402.759635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2402.792078] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 2403.796592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2403.820619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2403.869949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2404.176695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2405.216299] LNet: Removed LNI 192.168.202.29@tcp [ 2405.228744] LNet: Removed LNI 192.168.202.29@tcp2 [ 2405.298605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2405.357933] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2405.362900] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 2405.369974] LNet: 118169:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2405.914482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2406.214775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2407.264273] LNet: Removed LNI 192.168.202.29@tcp [ 2407.275098] LNet: Removed LNI 192.168.202.29@tcp2 [ 2407.294867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2407.355388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2407.378614] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2407.970272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2407.997625] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 2408.735955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2408.774903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2408.801625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2408.851094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2409.177400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2410.210925] LNet: Removed LNI 192.168.202.29@tcp [ 2411.238802] LNet: Removed LNI 192.168.202.29@tcp2 [ 2411.267386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2411.331398] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2411.337549] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 2411.785630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2418.336718] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2419.778954] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 01:37:34 (1788845854) [ 2420.193600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2420.248188] LNet: 119288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2420.251741] LNet: 119288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2420.270753] LNetError: 119288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.285802] LNetError: 119288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2420.296410] LNet: Removed LNI 192.168.202.29@tcp [ 2421.352967] LNet: Removed LNI 192.168.202.29@tcp2 [ 2421.386336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2421.727546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2421.767585] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2422.598447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2422.625354] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2423.443020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2423.461652] LNet: Removed LNI 10.1.2.3@tcp [ 2423.476759] LNet: Removed LNI 192.168.202.29@tcp [ 2423.492767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2423.558518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2423.585951] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2423.590986] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2424.045652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2424.062289] LNet: Removed LNI 192.168.202.29@tcp [ 2424.072388] LNet: Removed LNI 10.1.2.3@tcp [ 2431.316564] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 01:37:46 (1788845866) [ 2431.733449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2431.795426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2431.901187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2431.923090] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2432.739530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2432.770205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 2432.801748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 2432.861943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 2432.896200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 2432.934662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 2432.967599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 2433.033774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 2433.074432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 2433.124672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 2433.156582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.29@tcp [ 2433.279582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2440.320559] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 01:37:54 (1788845874) [ 2440.432788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2441.513553] LNet: Removed LNI 192.168.202.29@tcp [ 2441.564614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2441.723786] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2441.839011] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2441.851558] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2441.864813] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2444.244583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2444.280724] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2444.962076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2444.994398] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2446.963600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2447.015546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2447.093758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.29@tcp -m PUT -e local_timeout -r 1 [ 2447.184984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2447.243205] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2447.250737] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2447.532030] LNetError: 120963:0:(lib-move.c:4801:lnet_parse()) 192.168.202.29@tcp, src 192.168.202.29@tcp: Dropping PUT (error -108 looking up sender) [ 2448.552097] LNet: Removed LNI 10.1.2.3@tcp [ 2449.575935] LNet: Removed LNI 192.168.202.29@tcp [ 2450.043156] Key type .llcrypt unregistered [ 2450.046237] Key type ._llcrypt unregistered [ 2463.851690] Key type ._llcrypt registered [ 2463.858552] Key type .llcrypt registered [ 2465.205929] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 01:38:19 (1788845899) [ 2465.546508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2465.562316] alg: No test for adler32 (adler32-zlib) [ 2466.442805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2466.750200] LNet: 122271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2467.302367] Key type .llcrypt unregistered [ 2467.304650] Key type ._llcrypt unregistered [ 2474.526674] Key type ._llcrypt registered [ 2474.530246] Key type .llcrypt registered [ 2474.733434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2474.754180] alg: No test for adler32 (adler32-zlib) [ 2475.710677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2475.733770] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2483.274394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 2483.456835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2484.054163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.129@tcp [ 2484.269277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2484.293218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2484.320208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2484.348640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2484.380273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2484.406932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2484.441323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2484.474156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2484.487803] LNet: 122717:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2484.506914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2484.547720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2484.586423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2484.631768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2484.699349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2484.755328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2485.463278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2485.690477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2485.714640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2485.745635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2485.771878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2485.800702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2485.829973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2485.859885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2485.891628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2485.906844] LNet: 122835:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2485.923655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2485.959977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2485.988355] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.129@tcp [ 2485.993711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2486.023633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2486.056215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2486.088305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2486.786647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.129@tcp [ 2486.988531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2487.010304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2487.025493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2487.044848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2487.073111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2487.101158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2487.127321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2487.147726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2487.158776] LNet: 122952:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2487.171886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2487.186373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2487.202817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2487.221555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2487.237242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2487.252358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2487.649929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2487.888779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2487.920815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2487.952492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2487.984151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2488.009674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2488.039343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2488.083835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2488.155089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2488.216902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2488.276441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2488.320282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2488.343297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2488.372688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2488.390946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2488.555263] LNet: 123115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2488.558723] LNetError: 123115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2488.565833] LNet: Removed LNI 192.168.202.29@tcp [ 2488.889099] Key type .llcrypt unregistered [ 2488.890816] Key type ._llcrypt unregistered [ 2504.693510] Key type ._llcrypt registered [ 2504.695800] Key type .llcrypt registered [ 2505.986045] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 01:39:00 (1788845940) [ 2506.457110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2506.473611] alg: No test for adler32 (adler32-zlib) [ 2507.385389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2507.899472] LNet: 123999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2508.700632] Key type .llcrypt unregistered [ 2508.704176] Key type ._llcrypt unregistered [ 2515.557765] Key type ._llcrypt registered [ 2515.559967] Key type .llcrypt registered [ 2515.721857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2515.738721] alg: No test for adler32 (adler32-zlib) [ 2516.645960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2516.679451] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2523.900941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.129@tcp [ 2531.121237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2531.167263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2531.184562] LNet: Added LNI 192.168.202.29@tcp1 [8/256/0/180] [ 2531.964247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2532.388733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.129@tcp [ 2532.622554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2532.655502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2532.685325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2532.718744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2532.750612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2532.779733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2532.819479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2532.853192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2532.862907] LNet: 124648:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2532.872683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2532.891384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2532.913972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2532.935289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2532.962979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2532.990758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2533.610969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2533.948669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2533.982549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2534.040628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2534.076280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2534.116546] LNet: 2 peer NIs in recovery (showing 2): 192.168.202.129@tcp, 192.168.202.129@tcp1 [ 2534.163132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2534.207588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2534.246436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2534.271776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2534.286589] LNet: 124765:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2534.306548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2534.328702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2534.368404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2534.430923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2534.458668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2534.487990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2535.212677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.129@tcp [ 2535.538744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2535.566866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2535.592175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2535.623846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2535.657716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2535.686966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2535.722898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2535.776743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2535.802458] LNet: 124882:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2535.825950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2535.861432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2535.885650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2535.926134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2535.974225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2536.019881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2536.599183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2536.765462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2536.785107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2536.814783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2536.868470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2536.908094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2536.931746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2536.961807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2536.990478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2537.021223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2537.051346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2537.079211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2537.103802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2537.131422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2537.164798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2542.145615] LNet: 125080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2542.150968] LNetError: 125080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2542.163193] LNet: Removed LNI 192.168.202.29@tcp [ 2542.185082] LNet: Removed LNI 192.168.202.29@tcp1 [ 2542.745994] Key type .llcrypt unregistered [ 2542.751611] Key type ._llcrypt unregistered [ 2566.195458] Key type ._llcrypt registered [ 2566.198706] Key type .llcrypt registered [ 2568.271514] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 01:40:02 (1788846002) [ 2568.698960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2568.707631] alg: No test for adler32 (adler32-zlib) [ 2569.663852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2569.791736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2569.815720] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2570.613436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2570.637635] LNet: Added LNI 192.168.202.29@tcp2 [8/256/0/180] [ 2571.530632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.29@tcp [ 2571.709614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2572.768180] LNet: Removed LNI 192.168.202.29@tcp [ 2579.953419] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 01:40:14 (1788846014) [ 2581.911295] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2584.512843] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 01:40:18 (1788846018) [ 2592.724412] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 01:40:27 (1788846027) [ 2601.067842] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 01:40:35 (1788846035) [ 2608.873603] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 01:40:43 (1788846043) [ 2617.844577] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 01:40:52 (1788846052) [ 2625.634644] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 01:41:00 (1788846060) [ 2635.837478] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 01:41:10 (1788846070) [ 2644.924790] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 01:41:19 (1788846079) [ 2653.686420] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 01:41:28 (1788846088) [ 2662.021752] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 01:41:36 (1788846096) [ 2670.028761] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 01:41:44 (1788846104) [ 2676.905827] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 01:41:51 (1788846111) [ 2684.357551] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 01:41:59 (1788846119) [ 2692.345990] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 01:42:06 (1788846126) [ 2701.618879] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 01:42:15 (1788846135) [ 2711.043543] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 01:42:25 (1788846145) [ 2719.468735] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 01:42:34 (1788846154) [ 2726.939738] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 01:42:41 (1788846161) [ 2727.247652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2727.311480] LNet: 136777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2727.321667] LNetError: 136777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2728.367264] LNet: Removed LNI 192.168.202.29@tcp2 [ 2728.464217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2728.613262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2728.659694] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2729.984479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2730.019420] LNet: 136919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2730.026302] LNetError: 136919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2730.044474] LNet: Removed LNI 192.168.202.29@tcp [ 2730.081775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2730.153616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2730.187668] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2737.387533] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 01:42:52 (1788846172) [ 2745.756562] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 01:43:00 (1788846180) [ 2754.336604] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 01:43:08 (1788846188) [ 2761.702986] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 01:43:16 (1788846196) [ 2769.224777] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 01:43:24 (1788846204) [ 2776.341433] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 01:43:31 (1788846211) [ 2784.204928] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 01:43:39 (1788846219) [ 2791.369463] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 01:43:46 (1788846226) [ 2798.152256] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 01:43:53 (1788846233) [ 2805.853716] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 01:44:00 (1788846240) [ 2812.886464] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 01:44:07 (1788846247) [ 2820.194565] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 01:44:14 (1788846254) [ 2827.218058] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2581 sec ========== 01:44:21 (1788846261) [ 2827.530286] LNet: 144386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2827.534777] LNetError: 144386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2827.556163] LNet: Removed LNI 192.168.202.29@tcp [ 2828.249759] Key type .llcrypt unregistered [ 2828.253066] Key type ._llcrypt unregistered [ 2840.776659] Key type ._llcrypt registered [ 2840.782473] Key type .llcrypt registered [ 2841.311534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2841.328390] alg: No test for adler32 (adler32-zlib) [ 2842.604240] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2842.904208] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 2844.600262] Key type lgssc registered [ 2846.348786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2924.183268] Lustre: Mounted lustre-client - version 2.17.57_111_g33f3974 [ 2930.558579] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2940.398133] Lustre: Unmounted lustre-client [ 2959.244706] Key type lgssc unregistered [ 2959.470790] LNet: 147333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2959.479060] LNetError: 147333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2959.496859] LNet: Removed LNI 192.168.202.29@tcp [ 2960.301868] Key type .llcrypt unregistered [ 2960.304085] Key type ._llcrypt unregistered