[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 445886628 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002428] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006970] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009129] LSM: Security Framework initializing [ 0.010056] Yama: becoming mindful. [ 0.011039] SELinux: Initializing. [ 0.012067] *** VALIDATE selinux *** [ 0.020435] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025255] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026169] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027097] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028099] *** VALIDATE tmpfs *** [ 0.030242] *** VALIDATE proc *** [ 0.031212] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.034119] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035116] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038024] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042485] debug: unmapping init [mem 0xffffffff9a259000-0xffffffff9a260fff] [ 0.045184] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046610] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047021] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051019] ... max period: 00007fffffffffff [ 0.052019] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.055187] rcu: Hierarchical SRCU implementation. [ 0.057465] smp: Bringing up secondary CPUs ... [ 0.058629] x86: Booting SMP configuration: [ 0.059021] .... node #0, CPUs: #1 #2 #3 [ 0.062645] smp: Brought up 1 node, 4 CPUs [ 0.064016] smpboot: Max logical packages: 1 [ 0.065018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.250354] node 0 deferred pages initialised in 182ms [ 0.253142] devtmpfs: initialized [ 0.254285] x86/mm: Memory block size: 128MB [ 0.256694] gcov: version magic: 0x41383552 [ 0.258554] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.262148] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.266493] pinctrl core: initialized pinctrl subsystem [ 0.269235] [ 0.269984] ************************************************************* [ 0.272017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.275076] ** ** [ 0.277016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.280084] ** ** [ 0.282015] ** This means that this kernel is built to expose internal ** [ 0.285019] ** IOMMU data structures, which may compromise security on ** [ 0.287014] ** your system. ** [ 0.290019] ** ** [ 0.292015] ** If you see this message and you are not debugging the ** [ 0.295020] ** kernel, report this immediately to your vendor! ** [ 0.298019] ** ** [ 0.301020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.303016] ************************************************************* [ 0.306841] NET: Registered protocol family 16 [ 0.308469] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.311079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.313062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.316425] cpuidle: using governor menu [ 0.317784] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.320164] PCI: Using configuration type 1 for base access [ 0.322123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.330169] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.333034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.336208] cryptd: max_cpu_qlen set to 1000 [ 0.341326] ACPI: Added _OSI(Module Device) [ 0.343014] ACPI: Added _OSI(Processor Device) [ 0.344009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.345008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.348000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.354422] ACPI: Interpreter enabled [ 0.356064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.357010] ACPI: Using IOAPIC for interrupt routing [ 0.359122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.362397] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.373321] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.375042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.378015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.381067] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.385239] acpiphp: Slot [2] registered [ 0.387166] acpiphp: Slot [5] registered [ 0.388114] acpiphp: Slot [6] registered [ 0.389128] acpiphp: Slot [3] registered [ 0.391123] acpiphp: Slot [4] registered [ 0.392070] acpiphp: Slot [7] registered [ 0.394097] acpiphp: Slot [8] registered [ 0.395103] acpiphp: Slot [9] registered [ 0.396095] acpiphp: Slot [10] registered [ 0.398115] acpiphp: Slot [11] registered [ 0.399111] acpiphp: Slot [12] registered [ 0.401089] acpiphp: Slot [13] registered [ 0.402097] acpiphp: Slot [14] registered [ 0.404099] acpiphp: Slot [15] registered [ 0.405114] acpiphp: Slot [16] registered [ 0.407104] acpiphp: Slot [17] registered [ 0.408119] acpiphp: Slot [18] registered [ 0.410114] acpiphp: Slot [19] registered [ 0.412106] acpiphp: Slot [20] registered [ 0.413102] acpiphp: Slot [21] registered [ 0.415101] acpiphp: Slot [22] registered [ 0.417119] acpiphp: Slot [23] registered [ 0.418097] acpiphp: Slot [24] registered [ 0.420125] acpiphp: Slot [25] registered [ 0.422103] acpiphp: Slot [26] registered [ 0.423115] acpiphp: Slot [27] registered [ 0.424083] acpiphp: Slot [28] registered [ 0.426119] acpiphp: Slot [29] registered [ 0.427093] acpiphp: Slot [30] registered [ 0.429125] acpiphp: Slot [31] registered [ 0.431063] PCI host bridge to bus 0000:00 [ 0.432020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.434023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.437024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.439019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.441026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.444026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.446163] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.448529] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.452255] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.459548] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.464061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.467020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.469016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.473021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.475581] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.478890] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.481043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.484629] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.488938] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.498718] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.503015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.509866] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.518015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.524028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.547017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.556915] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.566014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.571018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.583019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.596773] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.599401] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.603364] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.606419] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.608223] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.612128] iommu: Default domain type: Passthrough [ 0.614433] SCSI subsystem initialized [ 0.616122] ACPI: bus type USB registered [ 0.617123] usbcore: registered new interface driver usbfs [ 0.619107] usbcore: registered new interface driver hub [ 0.622080] usbcore: registered new device driver usb [ 0.623168] pps_core: LinuxPPS API ver. 1 registered [ 0.625012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.628063] PTP clock support registered [ 0.630080] EDAC MC: Ver: 3.0.0 [ 0.631429] PCI: Using ACPI for IRQ routing [ 0.632601] NetLabel: Initializing [ 0.634013] NetLabel: domain hash size = 128 [ 0.636011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.639081] NetLabel: unlabeled traffic allowed by default [ 0.641137] vgaarb: loaded [ 0.644300] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.646023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.650268] clocksource: Switched to clocksource kvm-clock [ 0.754347] VFS: Disk quotas dquot_6.6.0 [ 0.756218] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.759122] *** VALIDATE ramfs *** [ 0.760625] *** VALIDATE hugetlbfs *** [ 0.762338] pnp: PnP ACPI init [ 0.764797] pnp: PnP ACPI: found 6 devices [ 0.786349] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.789417] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.791569] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.793551] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.795690] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.798468] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.800861] NET: Registered protocol family 2 [ 0.804368] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.809019] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.812225] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.817728] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.821428] TCP: Hash tables configured (established 65536 bind 65536) [ 0.824258] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.827305] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.830218] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.833105] NET: Registered protocol family 1 [ 0.835772] RPC: Registered named UNIX socket transport module. [ 0.838029] RPC: Registered udp transport module. [ 0.839783] RPC: Registered tcp transport module. [ 0.841598] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.844243] NET: Registered protocol family 44 [ 0.845868] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.848123] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.850349] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.852865] PCI: CLS 0 bytes, default 64 [ 0.854626] Unpacking initramfs... [ 2.239990] debug: unmapping init [mem 0xffff8b187cc64000-0xffff8b187ffcffff] [ 2.244885] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.247182] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.249908] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.748833] Initialise system trusted keyrings [ 2.751385] Key type blacklist registered [ 2.753596] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.763204] zbud: loaded [ 2.767201] *** VALIDATE nfs *** [ 2.768552] *** VALIDATE nfs4 *** [ 2.770249] pstore: using deflate compression [ 2.774436] Platform Keyring initialized [ 2.885609] NET: Registered protocol family 38 [ 2.887883] Key type asymmetric registered [ 2.889772] Asymmetric key parser 'x509' registered [ 2.891578] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.895417] io scheduler mq-deadline registered [ 2.897248] io scheduler kyber registered [ 2.899078] io scheduler bfq registered [ 2.901223] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.904191] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.907163] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.909674] ACPI: Power Button [PWRF] [ 2.914541] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.921439] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.934244] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.962066] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.991127] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.999338] Non-volatile memory driver v1.3 [ 3.000761] Linux agpgart interface v0.103 [ 3.033576] virtio_blk virtio1: [vda] 147224 512-byte logical blocks (75.4 MB/71.9 MiB) [ 3.036474] vda: detected capacity change from 0 to 75378688 [ 3.054821] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.058112] vdb: detected capacity change from 0 to 1073741824 [ 3.065522] libphy: Fixed MDIO Bus: probed [ 3.070739] usbcore: registered new interface driver usbserial_generic [ 3.073112] usbserial: USB Serial support registered for generic [ 3.075695] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.079845] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.081912] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.084344] mousedev: PS/2 mouse device common for all mice [ 3.087702] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.090132] rtc_cmos 00:05: RTC can wake from S4 [ 3.093777] rtc_cmos 00:05: registered as rtc0 [ 3.095689] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.098490] intel_pstate: CPU model not supported [ 3.099348] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.105545] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.105619] hid: raw HID events driver (C) Jiri Kosina [ 3.111344] usbcore: registered new interface driver usbhid [ 3.113067] usbhid: USB HID core driver [ 3.114527] drop_monitor: Initializing network drop monitor service [ 3.116646] Initializing XFRM netlink socket [ 3.118872] NET: Registered protocol family 10 [ 3.121673] Segment Routing with IPv6 [ 3.122951] NET: Registered protocol family 17 [ 3.124490] mpls_gso: MPLS GSO support [ 3.129579] RAS: Correctable Errors collector initialized. [ 3.131995] AVX version of gcm_enc/dec engaged. [ 3.133692] AES CTR mode by8 optimization enabled [ 3.216880] sched_clock: Marking stable (3216843672, 0)->(4092083997, -875240325) [ 3.219976] registered taskstats version 1 [ 3.223202] Loading compiled-in X.509 certificates [ 3.225273] zswap: loaded using pool lzo/zbud [ 3.252304] Key type big_key registered [ 3.267918] Key type encrypted registered [ 3.269711] ima: No TPM chip found, activating TPM-bypass! [ 3.272198] ima: Allocated hash algorithm: sha1 [ 3.274088] ima: No architecture policies found [ 3.276192] evm: Initialising EVM extended attributes: [ 3.278454] evm: security.selinux [ 3.279869] evm: security.ima [ 3.281229] evm: security.capability [ 3.282915] evm: HMAC attrs: 0x1 [ 3.286187] rtc_cmos 00:05: setting system clock to 2026-09-11 11:41:22 UTC (1789126882) [ 3.293918] debug: unmapping init [mem 0xffffffff9b203000-0xffffffff9b3fffff] [ 3.297303] debug: unmapping init [mem 0xffffffff99f82000-0xffffffff9a258fff] [ 3.308133] Write protecting the kernel read-only data: 28672k [ 3.311730] debug: unmapping init [mem 0xffffffff98603000-0xffffffff987fffff] [ 3.314904] debug: unmapping init [mem 0xffffffff98f14000-0xffffffff98ffffff] [ 3.348543] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.357987] systemd[1]: Detected virtualization kvm. [ 3.360161] systemd[1]: Detected architecture x86-64. [ 3.362842] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.390245] systemd[1]: No hostname configured. [ 3.391621] systemd[1]: Set hostname to . [ 3.393229] random: systemd: uninitialized urandom read (16 bytes read) [ 3.396027] systemd[1]: Initializing machine ID from random generator. [ 3.451730] random: ln: uninitialized urandom read (6 bytes read) [ 3.531585] random: systemd: uninitialized urandom read (16 bytes read) [ 3.534696] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.540946] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.548533] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.197903] device-mapper: uevent: version 1.0.3 [ 4.200513] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ 4.791378] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.963456] virtio_net virtio0 ens2: renamed from eth0 [ 5.105337] scsi host0: ata_piix [ 5.128180] scsi host1: ata_piix [ 5.129812] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.132457] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.229033] dracut-initqueue[592]: RTNETLINK answers: File exists [ 9.789049] random: crng init done [ 9.790680] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.121625] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.270977] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.527382] SELinux: Disabled at runtime. [ 11.591150] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.601993] systemd[1]: Detected virtualization kvm. [ 11.604209] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.114681] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.119421] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.125575] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.130312] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.133332] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.143190] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.150930] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ 12.187634] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.642722] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.919672] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.931853] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.012556] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.029867] EDAC sbridge: Ver: 1.1.2 [ 14.086207] Key type dns_resolver registered [ 14.383813] NFS: Registering the id_resolver key type [ 14.385968] Key type id_resolver registered [ 14.387869] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg630-client login: [ 47.417994] libcfs: loading out-of-tree module taints kernel. [ 47.555684] Key type ._llcrypt registered [ 47.563400] Key type .llcrypt registered [ 48.180919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 48.192823] alg: No test for adler32 (adler32-zlib) [ 49.451516] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 50.362272] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 52.127181] Key type lgssc registered [ 53.849082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 54.242182] hrtimer: interrupt took 1312451 ns [ 174.716455] Lustre: Mounted lustre-client - version 2.17.57_137_gfce00d1 [ 179.749281] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 200.160082] Lustre: lustre-OST0000-osc-ffff8b18d05b3000: disconnect after 23s idle [ 201.204077] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing check_logdir /tmp/testlogs/ [ 206.860877] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing yml_node [ 210.892453] Lustre: DEBUG MARKER: Client: 2.17.57.137 [ 213.229260] Lustre: DEBUG MARKER: MDS: 2.17.57.137 [ 215.628966] Lustre: DEBUG MARKER: OSS: 2.17.57.137 [ 216.865533] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Sep 11 07:44:54 EDT 2026 [ 231.254852] Lustre: Unmounted lustre-client [ 258.835286] Key type lgssc unregistered [ 259.048505] LNet: 8432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 259.055360] LNetError: 8432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 259.074932] LNet: Removed LNI 192.168.206.30@tcp [ 259.757153] Key type .llcrypt unregistered [ 259.759377] Key type ._llcrypt unregistered [ 283.887728] Key type ._llcrypt registered [ 283.891058] Key type .llcrypt registered [ 284.111342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 284.129995] alg: No test for adler32 (adler32-zlib) [ 285.117575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 285.213756] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 285.239843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 289.467910] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 289.668857] LNet: 9319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 289.683579] LNetError: 9319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 289.709887] LNet: Removed LNI 192.168.206.30@tcp [ 290.488687] Key type .llcrypt unregistered [ 290.490834] Key type ._llcrypt unregistered [ 296.618125] Key type ._llcrypt registered [ 296.625520] Key type .llcrypt registered [ 297.815255] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 07:46:15 (1789127175) [ 298.308280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 298.339888] alg: No test for adler32 (adler32-zlib) [ 299.268696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.325748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.339876] LNet: 9735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.354498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.391793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 299.404854] LNet: 9742:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 306.341411] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 07:46:24 (1789127184) [ 306.503837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.535671] LNet: 10319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.572174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.643592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 313.077192] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 07:46:30 (1789127190) [ 313.187561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.217340] LNet: 10901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.236893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.335917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 319.754796] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 07:46:37 (1789127197) [ 320.213463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 320.248177] LNet: 11490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 320.271179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.366707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 327.660719] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 07:46:44 (1789127204) [ 327.812381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.851535] LNet: 12069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.903711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.981434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 328.731691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 336.749899] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 07:46:54 (1789127214) [ 336.820894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.862934] LNet: 12660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.895874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.038087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 337.741279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 338.250572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 344.823828] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 07:47:02 (1789127222) [ 344.899248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.962455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.099793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 351.808912] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 07:47:09 (1789127229) [ 351.894135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.931856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.055985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 352.108787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 352.530280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 352.603514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 352.907565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 352.943753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 353.485308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 353.528226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 354.082536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 354.125434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 354.570402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 354.592594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 355.358224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 355.433775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 355.854597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 355.878801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 356.313432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 356.359871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 356.869407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 356.957471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 357.385392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 357.432542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 357.830963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 357.873638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 358.199832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 358.227956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 365.513218] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 07:47:22 (1789127242) [ 365.634128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.664278] LNet: 14527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.671283] LNet: 14527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 365.696917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.802383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 365.844297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 374.229480] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 07:47:31 (1789127251) [ 374.359606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.436596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.549785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 374.608562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 382.762985] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 07:47:40 (1789127260) [ 382.850903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.940598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.127702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 383.181917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 391.418475] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 07:47:48 (1789127268) [ 391.535724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.610650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.750361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 391.822876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 400.112500] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 07:47:57 (1789127277) [ 400.189081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.206582] LNet: 16871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.211233] LNet: 16871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 400.226406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.307183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 400.359789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 407.013393] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 07:48:04 (1789127284) [ 407.257528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.287457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.342432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 407.368383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 414.181208] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 07:48:11 (1789127291) [ 414.480661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.508504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.580831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 414.987867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 421.582355] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 07:48:19 (1789127299) [ 421.646700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.684574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.875323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 422.174420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 428.845371] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 07:48:26 (1789127306) [ 428.975880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.071529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.235678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 429.551659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 436.738377] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 07:48:34 (1789127314) [ 436.987383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.016676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.509478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 437.824754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 444.767318] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 07:48:42 (1789127322) [ 444.953606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.997269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.158281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 445.217402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 453.439839] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 07:48:51 (1789127331) [ 453.569574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.643731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.741589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 453.782890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.597971] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 07:48:58 (1789127338) [ 460.760907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.839177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.914219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 469.163508] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 07:49:06 (1789127346) [ 469.307952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.351182] LNet: 22313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.362188] LNet: 22313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 469.402418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.533869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 477.104828] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 07:49:14 (1789127354) [ 477.436139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.463209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.539876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 478.039849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 484.882920] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 07:49:22 (1789127362) [ 485.185793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.231068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.329949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 485.376533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 493.127503] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 07:49:30 (1789127370) [ 493.484425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.533266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.642184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 493.696727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 500.787187] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 07:49:38 (1789127378) [ 500.954608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.034345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.189040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 501.233529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 507.960693] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 07:49:45 (1789127385) [ 508.111978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.173146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.269830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 508.310284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 515.470975] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 07:49:53 (1789127393) [ 515.603230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.663481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.726575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 515.801240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 515.825381] LNetError: 25852:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 515.873407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 515.932855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 521.758264] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 07:49:59 (1789127399) [ 522.034919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.101335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.207518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.06P1aURGnM [ 522.286398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.06P1aURGnM [ 522.394506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.06P1aURGnM [ 528.043974] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 07:50:06 (1789127406) [ 528.334934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.417900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.467705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 528.507630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 528.556568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 535.043679] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 07:50:12 (1789127412) [ 535.215159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.267963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.342181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 535.367161] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 536.959122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 538.466062] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 540.375912] Key type lgssc registered [ 541.394179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 544.601583] LNetError: 27614:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.30@tcp99 has no available nets [ 547.460768] Key type lgssc unregistered [ 547.805967] LNetError: 28725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 548.844491] LNet: Removed LNI 192.168.206.30@tcp [ 549.423301] Key type .llcrypt unregistered [ 549.425803] Key type ._llcrypt unregistered [ 556.140667] Key type ._llcrypt registered [ 556.143421] Key type .llcrypt registered [ 557.213269] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 07:50:35 (1789127435) [ 557.815574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 557.841933] alg: No test for adler32 (adler32-zlib) [ 558.787589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.876489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 562.692575] LNet: Message forwarding will be enabled in 2s [ 565.791204] LNet: Message forwarding enabled [ 566.105804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 566.126896] LNet: Message forwarding will stop in 2s to 6s [ 570.650574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 571.632557] LNet: Message forwarding disabled [ 573.300292] LNet: Message forwarding will be enabled in 2s [ 576.799123] LNet: Message forwarding enabled [ 580.904885] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 07:50:58 (1789127458) [ 581.095199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.137518] LNet: 30129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.753182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.890501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 589.623559] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 07:51:07 (1789127467) [ 589.992110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.016714] LNet: 30715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.049461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.121051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 590.168721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 590.224809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 590.266603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 590.340247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 590.388747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 590.430289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 590.468347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 590.509539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 590.555632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 590.590483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 590.627992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 590.665812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 590.700287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 590.761212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 590.830913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 590.863966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 590.891735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 590.932604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 590.957546] LNetError: 30758:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 598.001483] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 07:51:15 (1789127475) [ 598.477282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.515518] LNet: 31330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 598.542193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.647488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 606.571458] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 07:51:23 (1789127483) [ 608.964836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg630-server [ 611.033422] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 07:51:28 (1789127488) [ 611.351272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.379281] LNet: 32261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.421911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.513557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 611.539566] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 612.533475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[31-35]@tcp [ 612.551705] LNet: 32394:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 612.610595] LNetError: 32265:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.31@tcp is being used as a gateway but routing feature is not turned on [ 612.627235] LNetError: 32265:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.31@tcp has gone from up to down [ 613.616035] LNetError: 32265:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.35@tcp is being used as a gateway but routing feature is not turned on [ 613.630457] LNetError: 32265:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 614.667089] LNetError: 32265:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.35@tcp is being used as a gateway but routing feature is not turned on [ 614.688960] LNetError: 32265:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 616.675685] LNetError: 32265:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.32@tcp is being used as a gateway but routing feature is not turned on [ 616.698137] LNetError: 32265:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 620.339452] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 07:51:37 (1789127497) [ 620.780766] LNetError: 32265:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.32@tcp is being used as a gateway but routing feature is not turned on [ 620.802418] LNetError: 32265:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 622.374514] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg630-server [ 624.311977] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 07:51:42 (1789127502) [ 624.537966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.568442] LNet: 33331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.577956] LNetError: 33331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 624.597459] LNet: Removed LNI 192.168.206.30@tcp [ 624.645058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.708433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 624.746529] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 625.547480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[31-36/2]@tcp [ 625.572457] LNet: 33448:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 625.588814] LNet: 33448:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 625.628658] LNetError: 33335:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.35@tcp has gone from up to down [ 625.635214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[31-36/2]@tcp [ 625.640179] LNetError: 33335:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 632.031621] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 07:51:49 (1789127509) [ 632.157471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.198083] LNetError: 34039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 632.231597] LNet: Removed LNI 192.168.206.30@tcp [ 632.260240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.378672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 632.438634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.479596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.543353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 632.637519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.679056] LNet: 34066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.683234] LNet: 34066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 632.712611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.765895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 632.896686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.958505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.056348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 633.161323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.212924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.317449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 633.495616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.548536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.609711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 633.689254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.726404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.812504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 633.905814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.938890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.998089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 634.111851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.152933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.212761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 634.280261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.331227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.450805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 634.585320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.644049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.720435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 634.801657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.870287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.984994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 641.721574] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 07:51:59 (1789127519) [ 643.999088] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg630-server [ 645.890331] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 07:52:03 (1789127523) [ 647.698292] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg630-server [ 649.667912] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 07:52:07 (1789127527) [ 649.789644] LNet: 35434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.794469] LNet: 35434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 650.790946] Key type .llcrypt unregistered [ 650.792930] Key type ._llcrypt unregistered [ 657.534312] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 657.675914] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 657.679557] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 657.689944] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 660.290215] Key type ._llcrypt registered [ 660.292191] Key type .llcrypt registered [ 660.448763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 660.465381] alg: No test for adler32 (adler32-zlib) [ 661.425216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.479621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 661.504657] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 662.088778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 662.105341] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 663.851205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 663.876858] LNet: Removed LNI 10.1.2.3@tcp [ 664.048361] LNet: 36163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.059700] LNetError: 36163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.078949] LNet: Removed LNI 192.168.206.30@tcp [ 664.856608] Key type .llcrypt unregistered [ 664.858903] Key type ._llcrypt unregistered [ 676.137864] Key type ._llcrypt registered [ 676.140133] Key type .llcrypt registered [ 677.310387] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 07:52:35 (1789127555) [ 678.000462] Key type .llcrypt unregistered [ 678.002818] Key type ._llcrypt unregistered [ 683.913434] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 684.048979] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 684.054933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 684.061808] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 686.565896] Key type ._llcrypt registered [ 686.570431] Key type .llcrypt registered [ 686.818983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 686.840713] alg: No test for adler32 (adler32-zlib) [ 687.738890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.814835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 687.865868] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 688.600766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 688.624827] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 691.410592] LNet: 37663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.415633] LNetError: 37663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 691.420178] LNet: Removed LNI 10.1.2.3@tcp [ 691.432312] LNet: Removed LNI 192.168.206.30@tcp [ 692.143293] Key type .llcrypt unregistered [ 692.145201] Key type ._llcrypt unregistered [ 704.439948] Key type ._llcrypt registered [ 704.442504] Key type .llcrypt registered [ 705.757305] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 07:53:03 (1789127583) [ 706.301552] Key type .llcrypt unregistered [ 706.305657] Key type ._llcrypt unregistered [ 713.184947] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 713.389749] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 713.397401] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 713.416621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 716.055235] Key type ._llcrypt registered [ 716.061516] Key type .llcrypt registered [ 716.246918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 716.290034] alg: No test for adler32 (adler32-zlib) [ 717.201758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.269066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 717.304842] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 719.335261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 719.358519] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 721.321190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 721.370158] LNet: Removed LNI 10.1.2.3@tcp [ 721.387269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 721.416027] LNet: Removed LNI 10.1.2.31@tcp [ 721.431292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 721.457860] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 722.125766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 722.146944] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 723.119681] LNet: 39593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.125779] LNetError: 39593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 723.133800] LNet: Removed LNI 10.1.2.31@tcp [ 723.152186] LNet: Removed LNI 10.1.2.3@tcp [ 723.870145] Key type .llcrypt unregistered [ 723.871959] Key type ._llcrypt unregistered [ 735.177154] Key type ._llcrypt registered [ 735.179327] Key type .llcrypt registered [ 736.232754] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 07:53:34 (1789127614) [ 736.766569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 736.781918] alg: No test for adler32 (adler32-zlib) [ 737.673522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.713135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 737.747111] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 738.500448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 738.517922] LNet: 40550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.526762] LNetError: 40550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 738.542676] LNet: Removed LNI 192.168.206.30@tcp [ 738.556807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.610874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.30@tcp [ 738.628834] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 739.376260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.30@tcp [ 739.396359] LNet: Removed LNI 192.168.206.30@tcp [ 739.472118] LNet: 40685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.192663] Key type .llcrypt unregistered [ 740.198686] Key type ._llcrypt unregistered [ 752.537890] Key type ._llcrypt registered [ 752.539951] Key type .llcrypt registered [ 753.984823] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 07:53:51 (1789127631) [ 756.664968] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg630-server [ 758.947947] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 07:53:56 (1789127636) [ 759.713698] Key type .llcrypt unregistered [ 759.715733] Key type ._llcrypt unregistered [ 766.994061] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 767.228592] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 767.233410] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 767.237821] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 769.966372] Key type ._llcrypt registered [ 769.970503] Key type .llcrypt registered [ 770.159268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 770.175342] alg: No test for adler32 (adler32-zlib) [ 771.125905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.204829] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 771.903319] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 773.839656] LNet: 42529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.853855] LNetError: 42529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.858417] LNet: Removed LNI 192.168.206.30@tcp1 [ 773.878094] LNet: Removed LNI 10.1.2.3@tcp2 [ 774.434527] Key type .llcrypt unregistered [ 774.437207] Key type ._llcrypt unregistered [ 787.422880] Key type ._llcrypt registered [ 787.425039] Key type .llcrypt registered [ 788.696318] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 07:54:26 (1789127666) [ 789.212117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 789.223593] alg: No test for adler32 (adler32-zlib) [ 790.219455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.306600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 790.329301] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 791.071184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.088862] LNet: 43501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.094613] LNetError: 43501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.115626] LNet: Removed LNI 192.168.206.30@tcp [ 791.136514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.214619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 791.245580] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 799.016782] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 07:54:36 (1789127676) [ 799.320778] LNet: 44093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.329831] LNetError: 44093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.354501] LNet: Removed LNI 192.168.206.30@tcp [ 799.949394] Key type .llcrypt unregistered [ 799.954451] Key type ._llcrypt unregistered [ 807.038938] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 807.211427] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 807.222184] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 807.232171] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 809.935867] Key type ._llcrypt registered [ 809.939681] Key type .llcrypt registered [ 810.117267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.135055] alg: No test for adler32 (adler32-zlib) [ 811.066410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.120638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.140748] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 811.683279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 811.704950] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 813.611225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.635979] LNet: 44839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.639742] LNetError: 44839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 813.643494] LNet: Removed LNI 10.1.2.3@tcp [ 813.655600] LNet: Removed LNI 192.168.206.30@tcp [ 813.671397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.711196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 813.732186] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 813.745497] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 814.156136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.174035] LNet: 44866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.187992] LNetError: 44866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 814.195669] LNetError: 44866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 814.208474] LNet: Removed LNI 10.1.2.3@tcp [ 814.219456] LNet: Removed LNI 192.168.206.30@tcp [ 814.257457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.319549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 814.340523] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 814.345937] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 814.806661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 814.830374] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 815.698566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.715098] LNet: 45028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.719686] LNetError: 45028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 815.727243] LNetError: 45028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 815.733373] LNet: Removed LNI 10.1.2.3@tcp [ 815.736233] LNet: Removed LNI 192.168.206.30@tcp [ 815.748253] LNet: Removed LNI 192.168.206.30@tcp2 [ 815.763705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.812335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 815.841114] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 815.846334] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 815.849537] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 822.617503] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 07:55:00 (1789127700) [ 823.116276] LNet: 45625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.129511] LNetError: 45625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 823.153658] LNetError: 45625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 823.165406] LNet: Removed LNI 10.1.2.3@tcp [ 823.168440] LNet: Removed LNI 192.168.206.30@tcp [ 823.196315] LNet: Removed LNI 192.168.206.30@tcp2 [ 824.046606] Key type .llcrypt unregistered [ 824.052688] Key type ._llcrypt unregistered [ 830.807541] Key type ._llcrypt registered [ 830.809052] Key type .llcrypt registered [ 831.021896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 831.058881] alg: No test for adler32 (adler32-zlib) [ 832.002332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.084438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 832.120444] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 833.004512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 833.021328] LNet: 46024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.028990] LNetError: 46024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.045453] LNet: Removed LNI 192.168.206.30@tcp [ 833.063574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.143243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 833.166495] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 833.550606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 833.565503] LNet: 46054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.572514] LNetError: 46054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.586903] LNet: Removed LNI 192.168.206.30@tcp [ 833.610612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.678140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 833.703857] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 834.272717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 834.315078] LNet: Removed LNI 192.168.206.30@tcp [ 834.335142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.425179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 834.450057] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 834.994386] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 835.076270] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 835.086100] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 835.097666] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 837.619060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 837.656527] LNet: 46159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.666922] LNet: 46159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 837.673224] LNetError: 46159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.681293] LNetError: 46159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 837.692959] LNet: Removed LNI 192.168.206.30@tcp [ 837.717201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.759903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 837.777622] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 844.458379] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 07:55:22 (1789127722) [ 844.654331] LNet: 46756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.662735] LNetError: 46756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.675057] LNet: Removed LNI 192.168.206.30@tcp [ 845.269803] Key type .llcrypt unregistered [ 845.271685] Key type ._llcrypt unregistered [ 852.814145] Key type ._llcrypt registered [ 852.818802] Key type .llcrypt registered [ 853.193221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 853.211561] alg: No test for adler32 (adler32-zlib) [ 854.176213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.263275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 854.303406] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 855.181229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 855.212930] LNet: 47159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.225718] LNetError: 47159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.246155] LNet: Removed LNI 192.168.206.30@tcp [ 855.267914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.351572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 855.384107] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 863.421157] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 07:55:41 (1789127741) [ 863.512177] LNet: 47742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.519332] LNetError: 47742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.542425] LNet: Removed LNI 192.168.206.30@tcp [ 864.099783] Key type .llcrypt unregistered [ 864.103885] Key type ._llcrypt unregistered [ 870.367274] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 870.557590] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 870.560358] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 870.573464] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 873.290837] Key type ._llcrypt registered [ 873.292805] Key type .llcrypt registered [ 873.610260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 873.628758] alg: No test for adler32 (adler32-zlib) [ 874.542940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.605241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 874.646736] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 876.020966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.060340] LNet: 48199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.083799] LNetError: 48199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.124763] LNet: Removed LNI 192.168.206.30@tcp [ 876.183321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.339339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 876.412974] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 884.431821] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 07:56:01 (1789127761) [ 884.558307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.603305] LNet: 48795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.616178] LNetError: 48795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.658769] LNet: Removed LNI 192.168.206.30@tcp [ 884.708678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.867350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 891.366645] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 07:56:09 (1789127769) [ 891.504497] LNet: 49374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.212759] Key type .llcrypt unregistered [ 892.215964] Key type ._llcrypt unregistered [ 899.528491] Key type ._llcrypt registered [ 899.530662] Key type .llcrypt registered [ 899.828841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 899.857698] alg: No test for adler32 (adler32-zlib) [ 900.806908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.900230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 900.936899] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 901.815509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.838367] LNet: 49780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.843367] LNetError: 49780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.861242] LNet: Removed LNI 192.168.206.30@tcp [ 901.875909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.923543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 901.942219] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 908.179888] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 07:56:26 (1789127786) [ 908.534276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.551153] LNet: 50375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.563188] LNetError: 50375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.583810] LNet: Removed LNI 192.168.206.30@tcp [ 908.614180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.679397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 915.289584] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 07:56:32 (1789127792) [ 916.879523] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 918.549716] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 07:56:36 (1789127796) [ 919.937818] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 922.184537] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 07:56:39 (1789127799) [ 923.939102] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 926.260543] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 07:56:43 (1789127803) [ 927.862898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 929.562978] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 07:56:47 (1789127807) [ 931.172160] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 933.044533] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 07:56:50 (1789127810) [ 933.276752] LNet: 52428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.894578] Key type .llcrypt unregistered [ 933.898263] Key type ._llcrypt unregistered [ 939.994790] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 940.119888] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 940.125710] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 940.130166] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 942.735809] Key type ._llcrypt registered [ 942.740568] Key type .llcrypt registered [ 942.871582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 942.880758] alg: No test for adler32 (adler32-zlib) [ 943.778141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.857493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 943.887545] LNet: Added LNI 192.168.206.30@tcp [8/512/0/180] [ 943.901434] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 943.906730] LNet: 52748:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 944.022636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.040352] LNet: 52767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.046827] LNetError: 52767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 944.055223] LNet: Removed LNI 10.1.2.3@tcp [ 944.058536] LNetError: 52767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.075566] LNet: Removed LNI 192.168.206.30@tcp [ 944.121423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.193938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 944.214261] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 944.275430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.298615] LNet: 52791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.303472] LNetError: 52791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.323836] LNet: Removed LNI 192.168.206.30@tcp [ 944.343310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.409030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 944.430567] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 944.434445] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 944.436908] LNet: 52799:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 944.508772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.524641] LNet: 52817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.532242] LNetError: 52817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 944.537340] LNet: Removed LNI 10.1.2.3@tcp [ 944.539561] LNetError: 52817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.557848] LNet: Removed LNI 192.168.206.30@tcp [ 944.588071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.663399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 944.700735] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 944.714261] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 944.717308] LNet: 52825:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 944.910705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.939573] LNet: 52851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.947312] LNetError: 52851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 944.956348] LNet: Removed LNI 10.1.2.3@tcp [ 944.959989] LNetError: 52851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.989916] LNet: Removed LNI 192.168.206.30@tcp [ 945.023729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.071780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 945.088539] LNet: Added LNI 192.168.206.30@tcp [8/256/0/182] [ 945.093923] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 945.174369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.197765] LNet: 52879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.202756] LNetError: 52879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 945.207776] LNet: Removed LNI 10.1.2.3@tcp [ 945.210856] LNetError: 52879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.225525] LNet: Removed LNI 192.168.206.30@tcp [ 945.250978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.322505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 945.349647] LNet: Added LNI 192.168.206.30@tcp [8/256/0/182] [ 945.353793] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 945.356622] LNet: 52887:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 945.415265] LNet: Removed LNI 10.1.2.3@tcp [ 945.429242] LNet: Removed LNI 192.168.206.30@tcp [ 946.014983] Key type .llcrypt unregistered [ 946.020276] Key type ._llcrypt unregistered [ 957.372377] Key type ._llcrypt registered [ 957.374817] Key type .llcrypt registered [ 958.508745] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 07:57:16 (1789127836) [ 959.990387] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 961.480644] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 07:57:19 (1789127839) [ 961.736991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 961.745954] alg: No test for adler32 (adler32-zlib) [ 962.750433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.828306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 962.855131] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 963.526358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.30 [ 963.550690] LNet: 54156:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 963.565640] LNet: Removed LNI 192.168.206.30@tcp [ 970.523801] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 07:57:27 (1789127847) [ 970.600750] LNet: 54717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.614407] LNetError: 54717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.640597] LNet: Removed LNI 192.168.206.30@tcp [ 971.491457] Key type .llcrypt unregistered [ 971.494670] Key type ._llcrypt unregistered [ 980.243521] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 980.424393] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 983.239675] Key type ._llcrypt registered [ 983.242369] Key type .llcrypt registered [ 983.503429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 983.534504] alg: No test for adler32 (adler32-zlib) [ 985.229239] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 993.099748] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 07:57:50 (1789127870) [ 993.309746] LNet: 55645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.317294] LNetError: 55645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 993.329947] LNet: Removed LNI 10.1.2.3@tcp [ 994.202581] Key type .llcrypt unregistered [ 994.209565] Key type ._llcrypt unregistered [ 1001.018298] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1001.272633] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1001.277664] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1001.284330] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1004.009000] Key type ._llcrypt registered [ 1004.010932] Key type .llcrypt registered [ 1004.090387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1004.108936] alg: No test for adler32 (adler32-zlib) [ 1005.031646] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1010.894848] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 07:58:08 (1789127888) [ 1012.647718] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1014.260555] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 07:58:12 (1789127892) [ 1014.515803] LNet: 56835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.530856] LNetError: 56835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1014.555586] LNet: Removed LNI 10.1.2.3@tcp [ 1015.127205] Key type .llcrypt unregistered [ 1015.131106] Key type ._llcrypt unregistered [ 1021.952127] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1022.214925] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1022.218650] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1022.230606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1025.042387] Key type ._llcrypt registered [ 1025.046120] Key type .llcrypt registered [ 1025.296530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1025.318144] alg: No test for adler32 (adler32-zlib) [ 1026.256598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1026.349132] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1034.858856] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 07:58:32 (1789127912) [ 1035.612388] LNet: 58071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.617746] LNetError: 58071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1035.629209] LNet: Removed LNI 10.1.2.3@tcp [ 1036.446479] Key type .llcrypt unregistered [ 1036.451256] Key type ._llcrypt unregistered [ 1042.938804] Key type ._llcrypt registered [ 1042.941231] Key type .llcrypt registered [ 1043.121804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1043.135525] alg: No test for adler32 (adler32-zlib) [ 1043.995463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1044.016239] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1052.715835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1053.646601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_interrupt -r 1 [ 1053.701500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_interrupt -r 1 [ 1053.737575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1053.764511] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1054.125533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1054.176285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1054.224741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1054.242682] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1054.282820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1054.333223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1054.385992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1054.423702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1054.460478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1054.487594] LNet: 58514:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1054.520499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1054.581340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1054.646510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1054.699547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1054.754787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1054.800686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1054.835983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1055.517252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_dropped -r 1 [ 1055.555965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_dropped -r 1 [ 1055.609900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1055.626885] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1055.639165] LNet: Skipped 1 previous similar message [ 1055.869931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1055.898744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1055.928685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1055.958484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1056.003584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1056.035878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1056.057312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1056.080090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1056.090983] LNet: 58623:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1056.110760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1056.131804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1056.151348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1056.179182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1056.222707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.260828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.317891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1056.800586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_aborted -r 1 [ 1056.828846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_aborted -r 1 [ 1056.857975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1056.870837] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1057.073729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1057.121089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1057.159327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1057.207329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1057.241975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1057.272330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1057.318326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1057.366017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1057.385703] LNet: 58732:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1057.405387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1057.451067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1057.496990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1057.541450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1057.591280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1057.629408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1057.663415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1058.115551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_no_route -r 1 [ 1058.164218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_no_route -r 1 [ 1058.212199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1058.431989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1058.465077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1058.506952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1058.533295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1058.565209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1058.590830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1058.605854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1058.623735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1058.640761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1058.663653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1058.704183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1058.746906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1058.769467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1058.803260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1058.834400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1059.400897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_timeout -r 1 [ 1059.433243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_timeout -r 1 [ 1059.473377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1059.495329] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1059.509940] LNet: Skipped 2 previous similar messages [ 1059.714104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1059.742882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1059.776796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1059.837790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1059.899204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1059.947583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1059.982746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1060.010330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1060.026389] LNet: 58951:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1060.032970] LNet: 58951:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1060.053933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1060.082566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1060.110389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1060.144879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1060.171549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1060.225387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1060.296181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1060.872796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_error -r 1 [ 1060.902975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_error -r 1 [ 1060.962584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1061.216758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1061.245536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1061.285549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1061.310073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1061.349189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1061.393337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1061.418915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1061.447760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1061.469943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1061.494565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1061.520789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1061.561449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1061.595677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1061.645343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1061.669702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1061.918978] LNet: 59105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.934310] LNetError: 59105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.950943] LNet: Removed LNI 192.168.206.30@tcp [ 1062.335397] Key type .llcrypt unregistered [ 1062.340165] Key type ._llcrypt unregistered [ 1079.136186] Key type ._llcrypt registered [ 1079.138904] Key type .llcrypt registered [ 1080.486983] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 07:59:18 (1789127958) [ 1081.914335] Key type .llcrypt unregistered [ 1081.917181] Key type ._llcrypt unregistered [ 1089.693595] Key type ._llcrypt registered [ 1089.700389] Key type .llcrypt registered [ 1090.147130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1090.172456] alg: No test for adler32 (adler32-zlib) [ 1091.296720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1091.354117] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1101.414591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1115.402744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1115.447867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1115.486559] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1117.381406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_interrupt -r 1 [ 1117.440462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_interrupt -r 1 [ 1117.486315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_interrupt -r 1 [ 1117.525198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_interrupt -r 1 [ 1117.554635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_interrupt -r 1 [ 1117.613971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_interrupt -r 1 [ 1117.662823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_interrupt -r 1 [ 1117.728748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_interrupt -r 1 [ 1117.776377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1117.796687] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1117.809635] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1118.060246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1118.084498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1118.113936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1118.150240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1118.214149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1118.260380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.309086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.379971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.421979] LNet: 60579:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1118.449126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1118.508372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.594653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1118.644720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.671593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.718547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1118.764822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1119.358588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_dropped -r 1 [ 1119.412736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_dropped -r 1 [ 1119.463879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_dropped -r 1 [ 1119.512711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_dropped -r 1 [ 1119.593867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_dropped -r 1 [ 1119.664887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_dropped -r 1 [ 1119.705862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_dropped -r 1 [ 1119.758814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_dropped -r 1 [ 1119.787788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1119.805992] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1119.812871] LNet: Skipped 2 previous similar messages [ 1120.184334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1120.233659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1120.278824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1120.323672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1120.353342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1120.412712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1120.460676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1120.511869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1120.543891] LNet: 60701:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1120.591580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.655477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1120.688157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.743921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.795092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.846368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.877217] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1120.888297] LNet: Skipped 3 previous similar messages [ 1120.903224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1121.694969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_aborted -r 1 [ 1121.747669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_aborted -r 1 [ 1121.847123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_aborted -r 1 [ 1121.919706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_aborted -r 1 [ 1121.974541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_aborted -r 1 [ 1122.042978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_aborted -r 1 [ 1122.098923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_aborted -r 1 [ 1122.160204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_aborted -r 1 [ 1122.218748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1122.571582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1122.642983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1122.689906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.741586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.805892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.838913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.870988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1122.896556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1122.916756] LNet: 60822:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1122.936728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.958979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.983785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1123.019854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1123.048686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.073306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1123.098628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1123.659951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_no_route -r 1 [ 1123.705968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_no_route -r 1 [ 1123.744694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_no_route -r 1 [ 1123.789839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_no_route -r 1 [ 1123.840569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_no_route -r 1 [ 1123.897684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_no_route -r 1 [ 1123.932491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_no_route -r 1 [ 1123.991613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_no_route -r 1 [ 1124.060868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1124.082935] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1124.093677] LNet: Skipped 4 previous similar messages [ 1124.352890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.390638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1124.436297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.472549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.508028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.542426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.571527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.613458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.647760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.687745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.704409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.732079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.761590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.825805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.861646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1125.500682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_timeout -r 1 [ 1125.531968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_timeout -r 1 [ 1125.589872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_timeout -r 1 [ 1125.672979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_timeout -r 1 [ 1125.725563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_timeout -r 1 [ 1125.787537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_timeout -r 1 [ 1125.845813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_timeout -r 1 [ 1125.898363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_timeout -r 1 [ 1125.942628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1126.213966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.265456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1126.329469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.382883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.430825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.459743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.501691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.553562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1126.576604] LNet: 61065:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1126.589138] LNet: 61065:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1126.607990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1126.636673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1126.680309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.708398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.736565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.773537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.806298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1127.322058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_error -r 1 [ 1127.381489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_error -r 1 [ 1127.438279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_error -r 1 [ 1127.507386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_error -r 1 [ 1127.559784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_error -r 1 [ 1127.641475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_error -r 1 [ 1127.675648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_error -r 1 [ 1127.729587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_error -r 1 [ 1127.784375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1128.052734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1128.104603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1128.147875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1128.189531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1128.218522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1128.248744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1128.280301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1128.303652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1128.352848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1128.388651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1128.427693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1128.458817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1128.486987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1128.517835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1128.561048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1134.529889] LNet: 61266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.543428] LNetError: 61266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.552278] LNet: Removed LNI 192.168.206.30@tcp [ 1135.601657] LNet: Removed LNI 192.168.206.30@tcp1 [ 1136.062609] Key type .llcrypt unregistered [ 1136.068294] Key type ._llcrypt unregistered [ 1153.082817] Key type ._llcrypt registered [ 1153.084848] Key type .llcrypt registered [ 1154.260441] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 08:00:32 (1789128032) [ 1155.194623] Key type .llcrypt unregistered [ 1155.201188] Key type ._llcrypt unregistered [ 1161.860722] Key type ._llcrypt registered [ 1161.863061] Key type .llcrypt registered [ 1162.225847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1162.275140] alg: No test for adler32 (adler32-zlib) [ 1163.198206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1163.254469] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1170.786517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1171.536853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_dropped -r 1 [ 1171.598655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_dropped -r 1 [ 1171.646818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1171.674797] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1171.959060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.006383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1172.056401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1172.127105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1172.181335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1172.227813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1172.306902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1172.350289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1172.369096] LNet: 62522:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1172.391480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1172.433653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1172.479476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.520403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1172.553941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.593468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.649706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1173.111979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_error -r 1 [ 1173.146401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_error -r 1 [ 1173.174991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1173.188240] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1173.420113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.458040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.492287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.523206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.559414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.587770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.624690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.660899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.679596] LNet: 62631:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.697794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.736474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.769222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.790947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.824769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.869075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.900345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1174.377853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_timeout -r 1 [ 1174.414156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_timeout -r 1 [ 1174.462824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1174.488269] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1174.719762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.742773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.770336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.800479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.828661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.852253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.881758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.904449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.916468] LNet: 62740:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.938447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.959233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.987625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.021988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.055442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.081185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.103318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1175.317156] LNet: 62785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.323483] LNetError: 62785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.339927] LNet: Removed LNI 192.168.206.30@tcp [ 1175.816519] Key type .llcrypt unregistered [ 1175.820677] Key type ._llcrypt unregistered [ 1192.301175] Key type ._llcrypt registered [ 1192.304424] Key type .llcrypt registered [ 1193.704167] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 08:01:11 (1789128071) [ 1194.747463] Key type .llcrypt unregistered [ 1194.751253] Key type ._llcrypt unregistered [ 1200.877773] Key type ._llcrypt registered [ 1200.882755] Key type .llcrypt registered [ 1201.221610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1201.253420] alg: No test for adler32 (adler32-zlib) [ 1202.323839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1202.385238] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1212.101829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1221.267677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1221.343327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1221.380448] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1223.333482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_dropped -r 1 [ 1223.398322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e remote_dropped -r 1 [ 1223.475165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_dropped -r 1 [ 1223.571611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e remote_dropped -r 1 [ 1223.606724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e remote_dropped -r 1 [ 1223.669292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e remote_dropped -r 1 [ 1223.717253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e remote_dropped -r 1 [ 1223.757204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e remote_dropped -r 1 [ 1223.794403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1223.825233] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1223.830687] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.130@tcp [ 1223.983452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1224.028846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1224.068229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1224.182968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1224.222944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1224.259166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1224.307805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1224.362897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1224.387271] LNet: 64271:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1224.419156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1224.455902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1224.515639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1224.557207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1224.599774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1224.643381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1224.699888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1224.774552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1225.709616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_error -r 1 [ 1225.775439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e remote_error -r 1 [ 1225.844819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_error -r 1 [ 1225.874092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e remote_error -r 1 [ 1225.902852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e remote_error -r 1 [ 1225.946864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e remote_error -r 1 [ 1226.000381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e remote_error -r 1 [ 1226.045834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e remote_error -r 1 [ 1226.096851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1226.126493] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1226.135766] LNet: Skipped 2 previous similar messages [ 1226.337710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1226.367052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1226.424591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.460467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.486737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1226.515507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1226.544204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1226.569444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.585779] LNet: 64394:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.602873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.637870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.689503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.741118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.782702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.839122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.886223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1226.915184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.737215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_timeout -r 1 [ 1227.840775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e remote_timeout -r 1 [ 1227.917511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_timeout -r 1 [ 1227.983334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e remote_timeout -r 1 [ 1228.044082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e remote_timeout -r 1 [ 1228.100655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e remote_timeout -r 1 [ 1228.149258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e remote_timeout -r 1 [ 1228.186679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e remote_timeout -r 1 [ 1228.219689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1228.261700] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1228.649287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1228.700299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1228.755705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.811857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.869913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.915741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.950407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1229.016324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1229.061892] LNet: 64518:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1229.082298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1229.111781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1229.195932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1229.257858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1229.326192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1229.393185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1229.427314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1229.482961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1234.738267] LNet: 64600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1234.773222] LNetError: 64600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1234.800884] LNet: Removed LNI 192.168.206.30@tcp [ 1234.849317] LNet: Removed LNI 192.168.206.30@tcp1 [ 1235.719720] Key type .llcrypt unregistered [ 1235.725398] Key type ._llcrypt unregistered [ 1252.510637] Key type ._llcrypt registered [ 1252.513969] Key type .llcrypt registered [ 1254.096600] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 08:02:11 (1789128131) [ 1256.042487] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1257.967269] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 08:02:15 (1789128135) [ 1258.981631] Key type .llcrypt unregistered [ 1258.986505] Key type ._llcrypt unregistered [ 1265.233295] Key type ._llcrypt registered [ 1265.236777] Key type .llcrypt registered [ 1265.450805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1265.469840] alg: No test for adler32 (adler32-zlib) [ 1266.421939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1266.483557] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1273.979887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1274.768805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e network_timeout -r 1 [ 1274.808528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e network_timeout -r 1 [ 1274.860521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1274.889522] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1275.177094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1275.238103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1275.286878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1275.336651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1275.374993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1275.425827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1275.453894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1275.477421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1275.496671] LNet: 66149:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1275.525712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1275.565683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1275.643827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1275.722313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1275.745069] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1275.763114] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1275.811714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1275.863696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1276.164147] LNet: 66192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1276.169749] LNetError: 66192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1276.186768] LNet: Removed LNI 192.168.206.30@tcp [ 1276.670507] Key type .llcrypt unregistered [ 1276.680548] Key type ._llcrypt unregistered [ 1297.485176] Key type ._llcrypt registered [ 1297.491346] Key type .llcrypt registered [ 1297.749807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1297.781817] alg: No test for adler32 (adler32-zlib) [ 1298.684744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1298.725785] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1306.472979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1315.821797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1315.886537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1315.925584] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1317.117640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e network_timeout -r 1 [ 1317.150382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e network_timeout -r 1 [ 1317.184058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e network_timeout -r 1 [ 1317.225700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e network_timeout -r 1 [ 1317.263947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e network_timeout -r 1 [ 1317.308460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e network_timeout -r 1 [ 1317.367489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e network_timeout -r 1 [ 1317.401566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e network_timeout -r 1 [ 1317.435580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1317.452459] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1317.670225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1317.712781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1317.770137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1317.800435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1317.830612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1317.856744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1317.892654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1317.923903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1317.941732] LNet: 67075:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1317.974287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1318.004385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1318.030830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1318.059660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1318.100682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1318.129645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1322.468613] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1322.920969] LNet: 67152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1322.930244] LNetError: 67152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1322.941547] LNet: Removed LNI 192.168.206.30@tcp [ 1322.956232] LNet: Removed LNI 192.168.206.30@tcp1 [ 1323.493805] Key type .llcrypt unregistered [ 1323.499180] Key type ._llcrypt unregistered [ 1341.216341] Key type ._llcrypt registered [ 1341.218990] Key type .llcrypt registered [ 1342.746363] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 08:03:40 (1789128220) [ 1343.245465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1343.257796] alg: No test for adler32 (adler32-zlib) [ 1344.269429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1344.321795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1344.347379] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1345.041528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1345.064675] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1346.042914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 1346.113633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1346.168714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1346.211397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1346.244659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1346.296543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1346.326522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.30@tcp [ 1346.349357] LNetError: 68254:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.30@tcp to recovery [ 1347.362311] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1347.362389] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1349.407387] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1349.963937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1353.504169] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1353.687498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1353.755633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1353.806948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1353.822624] LNet: 68361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1353.832585] LNetError: 68361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1353.842323] LNet: Removed LNI 192.168.206.30@tcp [ 1353.865157] LNet: Removed LNI 192.168.206.30@tcp1 [ 1353.886815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1353.963644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1353.991099] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1354.738965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1354.753345] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1355.700778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 1355.764971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1355.845598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1355.892648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1355.945174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1355.977465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1356.017995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1356.052475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.30@tcp [ 1356.074493] LNetError: 68633:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.30@tcp to recovery [ 1357.023379] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1359.813183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1363.167804] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1363.182067] LNet: Skipped 1 previous similar message [ 1368.096585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1368.164971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1368.191090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1374.626762] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 08:04:12 (1789128252) [ 1374.998056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1375.023665] LNet: 69351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1375.033570] LNetError: 69351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1375.049294] LNetError: 69351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1375.063950] LNet: Removed LNI 192.168.206.30@tcp [ 1375.086420] LNet: Removed LNI 192.168.206.30@tcp1 [ 1375.124691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1375.178312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1375.201906] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1375.802984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1375.819217] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1376.550207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 1376.603369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1376.645603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1376.678319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1376.709920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1376.734141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1376.757245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.30@tcp --health 0 [ 1376.834500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1377.185428] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.30@tcp [ 1377.189411] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1377.204689] LNet: Skipped 1 previous similar message [ 1392.282591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1392.409797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1392.443111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1392.476360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.30@tcp --health 500 [ 1392.592663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1399.281592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1399.351687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1399.375512] LNet: 69843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1399.384981] LNetError: 69843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1399.399732] LNetError: 69843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1399.407064] LNet: Removed LNI 192.168.206.30@tcp [ 1399.420519] LNet: Removed LNI 192.168.206.30@tcp1 [ 1399.442387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1399.496788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1399.529123] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1400.249609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1400.269549] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1401.017823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 1401.044405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1401.091311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1401.114489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1401.149488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1401.176563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1401.212148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1401.251524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.30@tcp --health 0 [ 1402.530697] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1402.540133] LNet: Skipped 3 previous similar messages [ 1403.798409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1413.084259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1413.133476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1413.177627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1419.442917] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 08:04:57 (1789128297) [ 1419.869591] LNet: 70890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1419.881940] LNetError: 70890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1419.888292] LNetError: 70890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1419.893371] LNet: Removed LNI 192.168.206.30@tcp [ 1419.908831] LNet: Removed LNI 192.168.206.30@tcp1 [ 1420.482929] Key type .llcrypt unregistered [ 1420.489528] Key type ._llcrypt unregistered [ 1425.823918] Key type ._llcrypt registered [ 1425.826408] Key type .llcrypt registered [ 1426.094204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1426.129962] alg: No test for adler32 (adler32-zlib) [ 1427.014480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1427.035479] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1435.774437] Lustre: DEBUG MARKER: Initial discovery [ 1436.100641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.130@tcp [ 1438.696984] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1438.749944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1438.819638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.130@tcp [ 1438.838793] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1438.864552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1438.896339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.130@tcp -r 1 -m GET -l 3 [ 1438.943461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.130@tcp -r 1 -m GET -e local_error [ 1438.977033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.130@tcp -r 1 -m PUT -l 6 [ 1439.327783] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1440.627858] Lustre: DEBUG MARKER: Force oleg630-client.virtnet to discover 192.168.206.130@tcp (in background) [ 1440.726089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.130@tcp [ 1442.309523] Lustre: DEBUG MARKER: Force oleg630-server to discover 192.168.206.30@tcp [ 1443.812496] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1444.924323] Lustre: DEBUG MARKER: Wait for 71648 [ 1446.321583] Lustre: DEBUG MARKER: Finished wait on 71648 [ 1446.668816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1446.703854] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1446.740142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1447.033134] LNet: 72109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.038285] LNetError: 72109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1447.066844] LNet: Removed LNI 192.168.206.30@tcp [ 1447.464661] Key type .llcrypt unregistered [ 1447.467972] Key type ._llcrypt unregistered [ 1463.785807] Key type ._llcrypt registered [ 1463.788607] Key type .llcrypt registered [ 1465.176819] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 08:05:42 (1789128342) [ 1466.135563] Key type .llcrypt unregistered [ 1466.138273] Key type ._llcrypt unregistered [ 1472.805992] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1472.881458] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1472.887785] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1472.890414] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1475.484688] Key type ._llcrypt registered [ 1475.490363] Key type .llcrypt registered [ 1475.724769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1475.745360] alg: No test for adler32 (adler32-zlib) [ 1476.678713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1476.732546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1476.755649] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1477.466898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1477.496971] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1485.892719] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 08:06:03 (1789128363) [ 1486.092718] LNet: 74186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1486.098693] LNetError: 74186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1486.107521] LNet: Removed LNI 10.1.2.3@tcp [ 1486.129931] LNet: Removed LNI 192.168.206.30@tcp [ 1486.961763] Key type .llcrypt unregistered [ 1486.973494] Key type ._llcrypt unregistered [ 1494.147825] Key type ._llcrypt registered [ 1494.150822] Key type .llcrypt registered [ 1494.289454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1494.303912] alg: No test for adler32 (adler32-zlib) [ 1495.178278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1495.223558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1495.242614] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1495.866435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1495.889562] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1496.804810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.30@tcp [ 1497.887268] LNetError: Unexpected error -22 connecting to 192.168.206.30@tcp at host 192.168.206.30:988 [ 1503.127867] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 08:06:20 (1789128380) [ 1503.302335] LNet: 75319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1503.312692] LNetError: 75319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1503.319311] LNet: Removed LNI 10.1.2.3@tcp [ 1503.330410] LNet: Removed LNI 192.168.206.30@tcp [ 1504.145681] Key type .llcrypt unregistered [ 1504.150476] Key type ._llcrypt unregistered [ 1510.349974] Key type ._llcrypt registered [ 1510.359023] Key type .llcrypt registered [ 1510.574857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1510.603376] alg: No test for adler32 (adler32-zlib) [ 1511.484167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.532838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1511.550298] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1512.101277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1512.115709] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 1512.828714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.30@tcp1 --nid 192.168.206.30@tcp2 [ 1516.329209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp1 192.168.206.30@tcp2 [ 1516.357389] LNetError: 76162:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.30@tcp1 to 192.168.206.30@tcp2 - Route table may be misconfigured [ 1516.387351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp2 192.168.206.30@tcp1 [ 1521.658570] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 08:06:39 (1789128399) [ 1521.788928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1521.804828] LNet: 76735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.814210] LNetError: 76735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1521.821984] LNet: Removed LNI 192.168.206.30@tcp1 [ 1522.866533] LNet: Removed LNI 192.168.206.30@tcp2 [ 1522.886352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.944551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1522.964300] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1523.703981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1523.727810] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1524.448359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 1524.483525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.30@tcp [ 1524.520680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp1 192.168.206.30@tcp1 [ 1524.556331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.30@tcp -d 192.168.206.30@tcp -e network_timeout [ 1524.594594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -e network_timeout [ 1524.624228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -e network_timeout [ 1524.654746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -e network_timeout [ 1524.704582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1524.724650] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1524.747675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1524.879707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1524.959105] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1524.977853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1530.542385] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 08:06:48 (1789128408) [ 1530.688602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1530.707444] LNet: 77598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.714230] LNetError: 77598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.717715] LNetError: 77598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1530.721851] LNet: Removed LNI 192.168.206.30@tcp [ 1531.751126] LNet: Removed LNI 192.168.206.30@tcp1 [ 1531.775262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1531.845150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1532.077190] LNet: 77639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1532.358667] Key type .llcrypt unregistered [ 1532.360081] Key type ._llcrypt unregistered [ 1542.403331] Key type ._llcrypt registered [ 1542.408404] Key type .llcrypt registered [ 1543.925274] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 08:07:01 (1789128421) [ 1544.315867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1544.326555] alg: No test for adler32 (adler32-zlib) [ 1545.217420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.479613] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1547.836587] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 08:07:05 (1789128425) [ 1548.122584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.147414] LNet: 78736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.172467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1548.244469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1548.271290] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1548.988751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1549.003903] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1549.778626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1549.820039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp1 [ 1549.852200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp1 [ 1555.260267] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 08:07:13 (1789128433) [ 1556.968154] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg630-server [ 1558.495359] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 08:07:16 (1789128436) [ 1560.458187] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg630-server [ 1562.030447] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 08:07:19 (1789128439) [ 1564.055868] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg630-server [ 1566.145981] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 08:07:23 (1789128443) [ 1568.321554] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg630-server [ 1570.457224] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 08:07:27 (1789128447) [ 1572.248291] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg630-server [ 1574.072704] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 08:07:31 (1789128451) [ 1576.114507] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg630-server [ 1577.665406] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 08:07:35 (1789128455) [ 1579.509485] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg630-server [ 1581.064387] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 08:07:39 (1789128459) [ 1583.062463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg630-server [ 1584.876992] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 08:07:42 (1789128462) [ 1587.060315] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg630-server [ 1589.119748] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 08:07:46 (1789128466) [ 1589.364492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.414361] LNet: 82613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.420076] LNetError: 82613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.425892] LNet: Removed LNI 192.168.206.30@tcp [ 1590.438455] LNet: Removed LNI 192.168.206.30@tcp1 [ 1590.478868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.536517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1590.560061] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1591.159791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1591.243460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1591.406746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.430480] LNet: 82761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.437413] LNetError: 82761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1591.443293] LNetError: 82761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1591.471170] LNet: Removed LNI 192.168.206.30@tcp [ 1591.528933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.633148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1591.676530] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1592.652948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1592.866890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.888314] LNetError: 82906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1593.995529] LNet: Removed LNI 192.168.206.30@tcp [ 1594.047416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.186188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1594.208457] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1594.869425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1594.976611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1595.278888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1595.308079] LNet: 83053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1595.320717] LNet: 83053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1595.327018] LNetError: 83053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1595.343932] LNet: Removed LNI 192.168.206.30@tcp [ 1595.376206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.463772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1595.506508] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1596.681559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1598.234401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.346388] LNet: Removed LNI 192.168.206.30@tcp [ 1599.438409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.523879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1599.560309] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1600.665809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1600.831706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1601.124806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.143963] LNet: 83355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.150299] LNet: 83355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1601.157810] LNetError: 83355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.163964] LNetError: 83355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1601.181994] LNet: Removed LNI 192.168.206.30@tcp [ 1601.213385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.280296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1601.306767] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1602.501604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1602.850855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.941699] LNet: Removed LNI 192.168.206.30@tcp [ 1603.981406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.054752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1604.093092] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1605.026485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1605.132993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1605.433096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1606.501835] LNet: Removed LNI 192.168.206.30@tcp [ 1606.555278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.623305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1606.653318] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1607.983882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1608.331074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.390488] LNet: Removed LNI 192.168.206.30@tcp [ 1609.464352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.607440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.657642] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1610.683190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1610.797114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1611.146878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.170968] LNet: 83941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.176259] LNet: 83941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1611.188089] LNetError: 83941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1611.206229] LNetError: 83941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1611.240179] LNet: Removed LNI 192.168.206.30@tcp [ 1611.268348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.341041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1611.367772] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1612.639904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1613.010698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1614.123331] LNet: Removed LNI 192.168.206.30@tcp [ 1614.148613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.269607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.306354] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1615.100499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1615.206080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1615.535741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1616.615335] LNet: Removed LNI 192.168.206.30@tcp [ 1616.651764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.735858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1616.776817] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1617.876646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1618.230190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.312431] LNet: Removed LNI 192.168.206.30@tcp [ 1619.369185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.471829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1619.495773] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1620.196756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1620.291682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.317809] LNet: Removed LNI 192.168.206.30@tcp [ 1620.341750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.411666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1620.433229] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1621.205666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1621.284113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.309547] LNet: Removed LNI 192.168.206.30@tcp [ 1621.339930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.377513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1621.433566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.458643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.503787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1621.571947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.603682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.641929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1621.705881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.746374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.842033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1621.875273] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1622.675480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.713338] LNet: Removed LNI 192.168.206.30@tcp [ 1622.731575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.793308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1629.961629] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 08:08:27 (1789128507) [ 1630.114685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.147992] LNet: 85397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1630.152024] LNet: 85397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1630.201790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.279884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1630.311887] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/180] [ 1631.453826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.484830] LNetError: 85539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.489700] LNetError: 85539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1631.505985] LNet: Removed LNI 192.168.206.30@tcp231 [ 1631.520846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.579827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1631.596858] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/180] [ 1631.604373] LNet: 85547:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1631.894265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1631.934304] LNet: Removed LNI 192.168.206.30@tcp231 [ 1631.963398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1631.995433] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/180] [ 1632.933866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1632.957899] LNet: Removed LNI 192.168.206.30@tcp231 [ 1632.983384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.034225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1633.057192] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/0] [ 1633.061369] LNet: 85700:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1633.580867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1633.626523] LNet: Removed LNI 192.168.206.30@tcp231 [ 1633.645712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1633.676509] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/0] [ 1634.657855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.694496] LNet: Removed LNI 192.168.206.30@tcp231 [ 1634.739521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.819367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1634.865941] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/60] [ 1634.870753] LNet: 85851:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1635.460475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1635.509455] LNet: Removed LNI 192.168.206.30@tcp231 [ 1635.541781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1635.601344] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/60] [ 1637.350051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.410713] LNet: Removed LNI 192.168.206.30@tcp231 [ 1637.432121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.497332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1637.522437] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/180] [ 1637.530498] LNet: 86003:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1638.013794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1638.065113] LNet: Removed LNI 192.168.206.30@tcp231 [ 1638.088726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1638.159527] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/180] [ 1646.427321] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 08:08:44 (1789128524) [ 1646.561809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.615632] LNet: Removed LNI 192.168.206.30@tcp231 [ 1646.682881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.744659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1646.761730] LNet: Added LNI 192.168.206.30@tcp232 [8/256/0/180] [ 1647.325709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1647.396619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1647.435914] LNet: Removed LNI 192.168.206.30@tcp232 [ 1647.454297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1647.501545] LNet: Added LNI 192.168.206.30@tcp232 [8/256/0/180] [ 1648.585646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1648.635488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1648.662778] LNet: Removed LNI 192.168.206.30@tcp232 [ 1648.695662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1648.736647] LNet: Added LNI 192.168.206.30@tcp232 [8/256/0/180] [ 1649.837533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1649.894751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1649.961571] LNet: Removed LNI 192.168.206.30@tcp232 [ 1650.010696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1650.077613] LNet: Added LNI 192.168.206.30@tcp232 [8/256/0/180] [ 1651.428911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1651.551244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1651.593649] LNet: Removed LNI 192.168.206.30@tcp232 [ 1651.655760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1659.191696] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 08:08:56 (1789128536) [ 1660.844463] Key type .llcrypt unregistered [ 1660.848343] Key type ._llcrypt unregistered [ 1668.566751] Key type ._llcrypt registered [ 1668.577038] Key type .llcrypt registered [ 1668.809193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1668.827783] alg: No test for adler32 (adler32-zlib) [ 1669.681775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1669.754996] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1677.101968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1685.951371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.992271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1686.026317] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1687.656239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_interrupt -r 2 [ 1687.693738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_interrupt -r 2 [ 1687.748581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_interrupt -r 2 [ 1687.796433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_interrupt -r 2 [ 1687.863751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_interrupt -r 2 [ 1687.927544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_interrupt -r 2 [ 1687.989159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_interrupt -r 2 [ 1688.045972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_interrupt -r 2 [ 1688.099321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1688.119925] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1688.124688] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1688.472489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1688.507595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1688.579940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1688.615804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1688.679657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1688.723905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1688.774263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1688.828377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1688.884989] LNet: 88511:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1688.915125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1688.975548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1689.023738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1689.077844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1689.119508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1689.181339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1689.947892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_dropped -r 2 [ 1690.007952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_dropped -r 2 [ 1690.060890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_dropped -r 2 [ 1690.148603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_dropped -r 2 [ 1690.224436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_dropped -r 2 [ 1690.286593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_dropped -r 2 [ 1690.333976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_dropped -r 2 [ 1690.386700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_dropped -r 2 [ 1690.425366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1690.464181] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1690.799765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1690.844124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1690.901216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1690.944343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1690.991357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1691.020582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1691.070801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1691.125870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1691.147463] LNet: 88631:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1691.169724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1691.197927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1691.231716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1691.257882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1691.307545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1691.373345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1692.038852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_aborted -r 2 [ 1692.076690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_aborted -r 2 [ 1692.112554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_aborted -r 2 [ 1692.145294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_aborted -r 2 [ 1692.172311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_aborted -r 2 [ 1692.229820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_aborted -r 2 [ 1692.316633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_aborted -r 2 [ 1692.373674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_aborted -r 2 [ 1692.436414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1692.472383] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1692.837994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1692.864327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1692.896846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1692.949500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1692.983577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1693.009345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1693.047581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1693.081226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1693.093621] LNet: 88751:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1693.114447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1693.141145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1693.188595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1693.243797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1693.299394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1693.351872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1694.116924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_no_route -r 2 [ 1694.147675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_no_route -r 2 [ 1694.173321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_no_route -r 2 [ 1694.209404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_no_route -r 2 [ 1694.242473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_no_route -r 2 [ 1694.288517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_no_route -r 2 [ 1694.347103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_no_route -r 2 [ 1694.422397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_no_route -r 2 [ 1694.453965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1694.491366] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1694.832828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1694.879322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1694.916589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1694.942406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1694.984806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1695.027639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1695.059579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1695.094517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1695.117131] LNet: 88872:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1695.134145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1695.160800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1695.188442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1695.215927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1695.245467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1695.268843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1695.757629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_timeout -r 2 [ 1695.788354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_timeout -r 2 [ 1695.816456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_timeout -r 2 [ 1695.857486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_timeout -r 2 [ 1695.890516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_timeout -r 2 [ 1695.919636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_timeout -r 2 [ 1695.952159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_timeout -r 2 [ 1695.980794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_timeout -r 2 [ 1696.010881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1696.309866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1696.335560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1696.368600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1696.402382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1696.441920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1696.470822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1696.519345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1696.574833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1696.622351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1696.651818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1696.672665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1696.720215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1696.762710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1696.786717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1697.315296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_dropped -r 2 [ 1697.375150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e remote_dropped -r 2 [ 1697.437247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_dropped -r 2 [ 1697.487956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e remote_dropped -r 2 [ 1697.534293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e remote_dropped -r 2 [ 1697.579363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e remote_dropped -r 2 [ 1697.611333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e remote_dropped -r 2 [ 1697.650288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e remote_dropped -r 2 [ 1697.683900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1697.706744] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.130@tcp [ 1697.995890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1698.038542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1698.085579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1698.124467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1698.185169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1698.236311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1698.285402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1698.332359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1698.360392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1698.425242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1698.502619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1698.587116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1698.648455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1698.693025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1704.661256] LNet: 89191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.699552] LNetError: 89191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.714810] LNet: Removed LNI 192.168.206.30@tcp [ 1704.732151] LNet: Removed LNI 192.168.206.30@tcp1 [ 1705.193819] Key type .llcrypt unregistered [ 1705.196243] Key type ._llcrypt unregistered [ 1721.692967] Key type ._llcrypt registered [ 1721.695990] Key type .llcrypt registered [ 1723.349515] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 08:10:00 (1789128600) [ 1723.867485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1723.883647] alg: No test for adler32 (adler32-zlib) [ 1724.830651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.896696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1724.914749] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1725.849191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1725.883172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.30@tcp [ 1725.901335] LNetError: 90170:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.30@tcp to recovery [ 1725.927184] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1725.980191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.30@tcp [ 1726.071656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1733.591939] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 08:10:11 (1789128611) [ 1733.874473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.927288] LNet: 90760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.940245] LNetError: 90760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.967877] LNet: Removed LNI 192.168.206.30@tcp [ 1733.984486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.042671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1734.073689] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1734.799105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1734.823922] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1735.784865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.30@tcp [ 1735.804755] LNetError: 91027:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.30@tcp to recovery [ 1736.097860] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1737.067620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.30@tcp [ 1744.501877] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 08:10:22 (1789128622) [ 1746.557963] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg630-server [ 1748.748567] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 08:10:26 (1789128626) [ 1750.782664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg630-server [ 1752.365713] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 08:10:30 (1789128630) [ 1754.263085] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg630-server [ 1755.709296] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 08:10:33 (1789128633) [ 1755.959093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.980144] LNet: 92650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.988348] LNetError: 92650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.999129] LNet: Removed LNI 192.168.206.30@tcp [ 1756.014498] LNet: Removed LNI 192.168.206.30@tcp1 [ 1756.040873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.115668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1756.155426] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1756.967263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1757.030326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1757.065983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1757.119493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1757.167428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1757.214706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1764.068833] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 08:10:42 (1789128642) [ 1764.226421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.239935] LNet: 93375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.243680] LNetError: 93375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.247927] LNetError: 93375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1764.261906] LNet: Removed LNI 192.168.206.30@tcp [ 1764.285872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.321849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1764.343954] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1770.483185] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 08:10:48 (1789128648) [ 1770.647474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.675957] LNet: 94076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.691770] LNetError: 94076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1770.724146] LNet: Removed LNI 192.168.206.30@tcp [ 1770.767950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.855371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1770.887879] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1777.303970] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 08:10:55 (1789128655) [ 1778.900281] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1780.456750] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 08:10:58 (1789128658) [ 1781.154955] LNet: 95113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.164890] LNetError: 95113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.204090] LNet: Removed LNI 192.168.206.30@tcp [ 1781.784410] Key type .llcrypt unregistered [ 1781.786685] Key type ._llcrypt unregistered [ 1787.369763] Key type ._llcrypt registered [ 1787.374107] Key type .llcrypt registered [ 1787.520267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1787.532418] alg: No test for adler32 (adler32-zlib) [ 1788.450395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1788.476916] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1795.235118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1800.104706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.130@tcp [ 1800.528825] LNet: 95538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1800.542247] LNetError: 95538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1800.567412] LNet: Removed LNI 192.168.206.30@tcp [ 1800.947645] Key type .llcrypt unregistered [ 1800.952423] Key type ._llcrypt unregistered [ 1810.785234] Key type ._llcrypt registered [ 1810.787234] Key type .llcrypt registered [ 1811.900400] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1813.376380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1814.964721] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 08:11:32 (1789128692) [ 1816.705860] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1818.296094] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 08:11:36 (1789128696) [ 1820.468841] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg630-server [ 1822.271898] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 08:11:40 (1789128700) [ 1824.093168] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg630-server [ 1826.027205] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 08:11:43 (1789128703) [ 1827.406195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1827.510378] alg: No test for adler32 (adler32-zlib) [ 1828.661861] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 1828.930142] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1830.583212] Key type lgssc registered [ 1832.083286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1854.540486] Key type lgssc unregistered [ 1854.781565] LNet: 98717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.793328] LNetError: 98717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1854.810764] LNet: Removed LNI 192.168.206.30@tcp [ 1855.450924] Key type .llcrypt unregistered [ 1855.452869] Key type ._llcrypt unregistered [ 1871.705724] Key type ._llcrypt registered [ 1871.708271] Key type .llcrypt registered [ 1873.395040] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 08:12:30 (1789128750) [ 1873.877542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1873.889404] alg: No test for adler32 (adler32-zlib) [ 1874.921362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1874.992082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1875.025777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1875.061106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1875.095524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1881.589851] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 08:12:39 (1789128759) [ 1881.705153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1881.774111] LNet: 100126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1881.821205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.950534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1881.976805] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1883.031797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1883.069036] LNet: 100275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.082633] LNetError: 100275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.107472] LNet: Removed LNI 192.168.206.30@tcp [ 1883.140698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1883.242525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1883.279720] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1884.037356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1884.081290] LNet: 100304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.092485] LNetError: 100304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.128496] LNet: Removed LNI 192.168.206.30@tcp [ 1884.179848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.300959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1884.386966] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1893.258798] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 08:12:50 (1789128770) [ 1893.381522] LNet: 100886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.399033] LNetError: 100886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.450373] LNet: Removed LNI 192.168.206.30@tcp [ 1894.484608] Key type .llcrypt unregistered [ 1894.490491] Key type ._llcrypt unregistered [ 1902.973410] Key type ._llcrypt registered [ 1902.977136] Key type .llcrypt registered [ 1903.225366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1903.263335] alg: No test for adler32 (adler32-zlib) [ 1904.209592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1904.285528] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1904.289965] LNet: *** cfs_fail_loc=e003, val=100*** [ 1905.418709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1905.467635] LNet: *** cfs_fail_loc=e003, val=100*** [ 1905.474826] LNet: Skipped 2 previous similar messages [ 1905.489581] Ping reply received with mapping: [ 1905.493043] found metadata reply NI[192.168.206.30@tcp] [ 1905.500059] 1e ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1905.510614] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1905.518723] 5a 5a 5a 5a 5a 5a 5a 5a [ 1905.843992] LNet: 101306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.849256] LNetError: 101306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.864056] LNet: Removed LNI 192.168.206.30@tcp [ 1906.843484] Key type .llcrypt unregistered [ 1906.848431] Key type ._llcrypt unregistered [ 1918.628990] Key type ._llcrypt registered [ 1918.631151] Key type .llcrypt registered [ 1919.958368] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 08:13:17 (1789128797) [ 1920.525381] Key type .llcrypt unregistered [ 1920.529034] Key type ._llcrypt unregistered [ 1927.233698] Key type ._llcrypt registered [ 1927.238155] Key type .llcrypt registered [ 1927.505886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1927.533152] alg: No test for adler32 (adler32-zlib) [ 1928.666387] LNetError: 102363:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1929.035413] Key type .llcrypt unregistered [ 1929.038581] Key type ._llcrypt unregistered [ 1933.773140] Key type ._llcrypt registered [ 1933.776856] Key type .llcrypt registered [ 1935.284503] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 08:13:32 (1789128812) [ 1936.093354] Key type .llcrypt unregistered [ 1936.097892] Key type ._llcrypt unregistered [ 1943.022874] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1943.035401] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1945.651594] Key type ._llcrypt registered [ 1945.655959] Key type .llcrypt registered [ 1945.873468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1945.890709] alg: No test for adler32 (adler32-zlib) [ 1946.875485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.943968] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1956.853843] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 08:13:54 (1789128834) [ 1957.043894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1957.077774] LNet: 104150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1957.087889] LNetError: 104150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1957.117627] LNet: Removed LNI 10.1.2.3@tcp [ 1957.135781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1957.909523] Key type .llcrypt unregistered [ 1957.911842] Key type ._llcrypt unregistered [ 1965.627137] Key type ._llcrypt registered [ 1965.628327] Key type .llcrypt registered [ 1965.758302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1965.784668] alg: No test for adler32 (adler32-zlib) [ 1966.697063] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1966.781937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1967.036548] LNet: 104444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.053310] LNetError: 104444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.073258] LNet: Removed LNI 192.168.206.30@tcp [ 1968.359779] Key type .llcrypt unregistered [ 1968.361509] Key type ._llcrypt unregistered [ 1981.612554] Key type ._llcrypt registered [ 1981.614438] Key type .llcrypt registered [ 1983.243646] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 08:14:20 (1789128860) [ 1984.052422] Key type .llcrypt unregistered [ 1984.054615] Key type ._llcrypt unregistered [ 1994.538581] Key type ._llcrypt registered [ 1994.540777] Key type .llcrypt registered [ 1994.703808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1994.731134] alg: No test for adler32 (adler32-zlib) [ 2004.094510] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 08:14:41 (1789128881) [ 2004.442962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2004.486786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2004.522938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2004.537497] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2005.242939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2005.297059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2005.333459] LNet: Removed LNI 192.168.206.30@tcp [ 2005.592962] LNet: 106203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.013050] Key type .llcrypt unregistered [ 2006.017401] Key type ._llcrypt unregistered [ 2017.688620] Key type ._llcrypt registered [ 2017.692392] Key type .llcrypt registered [ 2018.899595] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 08:14:56 (1789128896) [ 2019.714165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2019.744961] alg: No test for adler32 (adler32-zlib) [ 2020.645757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2020.716710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2020.740970] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2021.455141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2021.483360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.30@tcp [ 2021.502370] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2021.508731] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2021.512913] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2021, 0, 0 [ 2021.521155] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2021.527475] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2021.534051] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2021.539501] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2021, 0, 0 [ 2021.542676] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2021.548133] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2021.553765] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2021.559079] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2021, 0, 2071 [ 2021.565336] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2021.568364] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2021.572985] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2021.578339] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2021, 0, 2071 [ 2021.585153] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2027.719387] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 08:15:05 (1789128905) [ 2028.500629] LNet: 107743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.516849] LNetError: 107743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.534369] LNet: Removed LNI 192.168.206.30@tcp [ 2029.397460] Key type .llcrypt unregistered [ 2029.399076] Key type ._llcrypt unregistered [ 2036.594447] Key type ._llcrypt registered [ 2036.598709] Key type .llcrypt registered [ 2036.854416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2036.910303] alg: No test for adler32 (adler32-zlib) [ 2037.812415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2037.847770] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2046.111306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 2055.655941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2055.719258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2055.771850] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 2056.880824] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2057.015432] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2057.033457] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2057.042931] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2059.595201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2059.622667] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2070.230681] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 08:15:48 (1789128948) [ 2070.417537] LNet: 109252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.422864] LNetError: 109252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2070.429088] LNet: Removed LNI 192.168.206.30@tcp [ 2070.431449] LNet: Removed LNI 192.168.206.30@tcp1 [ 2070.447553] LNet: Removed LNI 10.1.2.3@tcp99 [ 2071.062754] Key type .llcrypt unregistered [ 2071.064725] Key type ._llcrypt unregistered [ 2077.898511] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2077.904365] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2080.700803] Key type ._llcrypt registered [ 2080.707650] Key type .llcrypt registered [ 2081.135506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2081.181243] alg: No test for adler32 (adler32-zlib) [ 2082.135566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2082.209224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2082.244237] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2083.261745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2083.278961] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2085.755218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2085.828726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2085.873435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2085.901275] LNetError: 110005:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2085.922853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2085.971409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2086.538306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2086.636718] LNet: 110025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.645497] LNetError: 110025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2086.650910] LNet: Removed LNI 10.1.2.3@tcp [ 2087.719194] LNet: Removed LNI 192.168.206.30@tcp [ 2088.413212] Key type .llcrypt unregistered [ 2088.415297] Key type ._llcrypt unregistered [ 2103.125896] Key type ._llcrypt registered [ 2103.130163] Key type .llcrypt registered [ 2105.029451] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 08:16:22 (1789128982) [ 2105.454643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2105.492293] alg: No test for adler32 (adler32-zlib) [ 2106.506671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.601429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2106.649478] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2107.594869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2107.674195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg630-client@tcp [ 2114.425538] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 08:16:32 (1789128992) [ 2114.805353] LNet: 111563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2114.814197] LNetError: 111563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2114.828975] LNet: Removed LNI 192.168.206.30@tcp [ 2115.477357] Key type .llcrypt unregistered [ 2115.480241] Key type ._llcrypt unregistered [ 2122.737504] Key type ._llcrypt registered [ 2122.742214] Key type .llcrypt registered [ 2122.888266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2122.898185] alg: No test for adler32 (adler32-zlib) [ 2123.770807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2123.812594] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2123.871230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2130.315236] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 08:16:48 (1789129008) [ 2131.753253] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2133.383938] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 08:16:51 (1789129011) [ 2135.005207] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2137.367432] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 08:16:54 (1789129014) [ 2137.496390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.549309] LNet: 112979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2137.562771] LNetError: 112979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.591395] LNet: Removed LNI 192.168.206.30@tcp [ 2137.615607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.715502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2137.748979] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2138.549810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2138.617189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2145.038422] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 08:17:02 (1789129022) [ 2145.261338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2145.310512] LNet: 113694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2145.324349] LNetError: 113694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2145.342449] LNet: Removed LNI 192.168.206.30@tcp [ 2145.374325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.444095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2145.485740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2146.149267] Key type .llcrypt unregistered [ 2146.151112] Key type ._llcrypt unregistered [ 2157.506792] Key type ._llcrypt registered [ 2157.511508] Key type .llcrypt registered [ 2158.834946] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 08:17:16 (1789129036) [ 2159.354972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2159.376962] alg: No test for adler32 (adler32-zlib) [ 2160.318977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2160.400195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2160.440635] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2161.308247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2161.377981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2167.566291] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 08:17:25 (1789129045) [ 2167.665119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2167.691525] LNet: 115249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2167.707148] LNetError: 115249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2167.732797] LNet: Removed LNI 192.168.206.30@tcp [ 2167.793263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2167.848786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2167.915471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2174.833555] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 08:17:32 (1789129052) [ 2175.222680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2175.252477] LNet: 115835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2175.286094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2175.394642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2175.918940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2175.958891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2176.042230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2176.534838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2176.584877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2176.674489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2176.715206] LNet: 115868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2176.732632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2176.859952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2177.546326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2177.606310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2177.681421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2178.250878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2178.309887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2186.041708] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 08:17:43 (1789129063) [ 2186.449840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2186.510812] LNet: 116469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2186.565431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.724412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2186.790243] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2187.861272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2187.893754] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2189.264319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2189.371035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2189.783883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.837120] LNetError: 116748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2189.858237] LNet: Removed LNI 192.168.206.30@tcp [ 2189.894976] LNet: Removed LNI 192.168.206.30@tcp2 [ 2189.920281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.994924] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2190.000869] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2190.005187] LNet: 116754:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2190.698205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2191.015174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2191.035547] LNet: 116791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.047042] LNet: 116791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2191.053112] LNetError: 116791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2191.057761] LNetError: 116791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2191.062480] LNet: Removed LNI 192.168.206.30@tcp [ 2191.071392] LNet: Removed LNI 192.168.206.30@tcp2 [ 2191.093441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2191.131579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2191.170475] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2192.171280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2192.198743] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2193.534667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2193.558945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2193.604551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2193.783381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2193.801414] LNetError: 117076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.805055] LNetError: 117076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2194.847643] LNet: Removed LNI 192.168.206.30@tcp [ 2195.890393] LNet: Removed LNI 192.168.206.30@tcp2 [ 2195.956092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2196.063279] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2196.080655] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2196.088091] LNet: 117082:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2196.554859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2196.737871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2196.762959] LNet: Removed LNI 192.168.206.30@tcp [ 2197.799861] LNet: Removed LNI 192.168.206.30@tcp2 [ 2197.898944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2197.980258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2198.015147] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2199.151994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2199.183576] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2200.285755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2200.360819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2200.527373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2200.559164] LNet: 117399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.573928] LNet: 117399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2200.580237] LNetError: 117399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2200.591798] LNetError: 117399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2200.597545] LNet: Removed LNI 192.168.206.30@tcp [ 2200.614238] LNet: Removed LNI 192.168.206.30@tcp2 [ 2200.641458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2200.778980] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2200.786530] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2200.792490] LNet: 117405:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2201.792882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2202.144601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2203.232683] LNet: Removed LNI 192.168.206.30@tcp [ 2203.255417] LNet: Removed LNI 192.168.206.30@tcp2 [ 2203.340193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.475145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2203.535701] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2204.739996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2204.758937] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2206.026731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2206.066342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2206.120417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2206.321569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2207.391141] LNet: Removed LNI 192.168.206.30@tcp [ 2208.422159] LNet: Removed LNI 192.168.206.30@tcp2 [ 2208.501488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2208.686444] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2208.692681] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2208.699954] LNet: 117733:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2209.103786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2209.276819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2209.296891] LNetError: 117770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2209.302140] LNetError: 117770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2209.308500] LNet: Removed LNI 192.168.206.30@tcp [ 2210.343580] LNet: Removed LNI 192.168.206.30@tcp2 [ 2210.401771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.510570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2210.590116] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2211.775351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2211.809104] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2212.506365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2212.585132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2212.757063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2213.392907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2214.498213] LNet: Removed LNI 192.168.206.30@tcp [ 2214.512802] LNet: Removed LNI 192.168.206.30@tcp2 [ 2214.549444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.693149] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2214.716764] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2214.728328] LNet: 118073:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2215.522625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2216.103727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2217.183209] LNet: Removed LNI 192.168.206.30@tcp [ 2217.230092] LNet: Removed LNI 192.168.206.30@tcp2 [ 2217.289507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.340473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2217.377705] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2218.015791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2218.032762] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2218.905621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2218.937870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2218.963945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2219.015869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2219.342828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2219.359682] LNet: 118426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.364319] LNet: 118426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2220.383364] LNet: Removed LNI 192.168.206.30@tcp [ 2221.413586] LNet: Removed LNI 192.168.206.30@tcp2 [ 2221.485888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2221.563633] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2221.571327] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2222.302730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2230.514399] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2232.334611] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 08:18:30 (1789129110) [ 2232.741668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2232.770922] LNetError: 119193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2232.777749] LNetError: 119193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2232.783910] LNet: Removed LNI 192.168.206.30@tcp [ 2233.830253] LNet: Removed LNI 192.168.206.30@tcp2 [ 2233.923163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2234.342758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2234.375443] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2235.165689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2235.193582] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2236.224755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2236.254080] LNet: Removed LNI 10.1.2.3@tcp [ 2236.265585] LNet: Removed LNI 192.168.206.30@tcp [ 2236.284795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2236.383984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2236.422647] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2236.430350] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2236.959903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2237.007481] LNet: Removed LNI 192.168.206.30@tcp [ 2237.021553] LNet: Removed LNI 10.1.2.3@tcp [ 2244.567359] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 08:18:42 (1789129122) [ 2244.818283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2244.882266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.947829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2244.965105] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2245.814979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2245.856697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2245.888561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2245.930916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2245.970149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2245.991301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2246.039924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2246.070302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2246.110470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2246.156297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2246.177980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2246.288399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2252.417685] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 08:18:50 (1789129130) [ 2252.741544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2252.772606] LNet: 120807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.781359] LNet: 120807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2253.871703] LNet: Removed LNI 192.168.206.30@tcp [ 2253.886536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.000273] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2254.127366] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2254.133052] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2254.137675] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2256.444842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2256.463641] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2257.121293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2257.159840] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2259.372846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2259.405515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2259.475347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.30@tcp -m PUT -e local_timeout -r 1 [ 2259.542701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2259.611052] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2259.615060] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2259.861310] LNetError: 120872:0:(lib-move.c:4801:lnet_parse()) 192.168.206.30@tcp, src 192.168.206.30@tcp: Dropping PUT (error -108 looking up sender) [ 2260.899464] LNet: Removed LNI 10.1.2.3@tcp [ 2261.931614] LNet: Removed LNI 192.168.206.30@tcp [ 2262.381369] Key type .llcrypt unregistered [ 2262.386526] Key type ._llcrypt unregistered [ 2274.729775] Key type ._llcrypt registered [ 2274.732396] Key type .llcrypt registered [ 2276.040525] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 08:19:13 (1789129153) [ 2276.451424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2276.483661] alg: No test for adler32 (adler32-zlib) [ 2277.467908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2277.967780] LNet: 122188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2278.619277] Key type .llcrypt unregistered [ 2278.621553] Key type ._llcrypt unregistered [ 2284.701241] Key type ._llcrypt registered [ 2284.703494] Key type .llcrypt registered [ 2284.984409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2285.006639] alg: No test for adler32 (adler32-zlib) [ 2285.955418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2285.976990] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2293.607938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 2293.829402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2294.572474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.130@tcp [ 2294.774436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2294.809040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2294.852192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2294.891433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2294.954602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2294.999228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2295.051422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2295.099608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2295.119346] LNet: 122641:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2295.138481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2295.177721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2295.199308] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.130@tcp [ 2295.227251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2295.255659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2295.301128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2295.337401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2295.898397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2296.170180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2296.211338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2296.255797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2296.296485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2296.337307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2296.371450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2296.422868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2296.456680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2296.479967] LNet: 122759:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2296.502540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2296.537406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2296.591100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2296.628452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2296.668169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2296.724969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2297.415386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.130@tcp [ 2297.719502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2297.756566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2297.789954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2297.819488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2297.845946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2297.880620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2297.908905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2297.943184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2297.957950] LNet: 122876:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2297.974474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2298.016468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2298.060559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2298.120557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2298.183433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2298.232060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2298.845418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2299.125962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2299.164411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2299.194826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2299.229974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2299.270798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2299.313876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2299.365983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2299.430257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2299.472489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2299.519467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2299.549238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2299.575580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2299.602586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2299.631925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2300.013573] LNet: 123043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2300.023121] LNetError: 123043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2300.046174] LNet: Removed LNI 192.168.206.30@tcp [ 2300.401796] Key type .llcrypt unregistered [ 2300.405881] Key type ._llcrypt unregistered [ 2316.849190] Key type ._llcrypt registered [ 2316.856205] Key type .llcrypt registered [ 2318.122146] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 08:19:55 (1789129195) [ 2318.469318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2318.487722] alg: No test for adler32 (adler32-zlib) [ 2319.456439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.989269] LNet: 123931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2320.826399] Key type .llcrypt unregistered [ 2320.829783] Key type ._llcrypt unregistered [ 2329.467786] Key type ._llcrypt registered [ 2329.472875] Key type .llcrypt registered [ 2329.746609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2329.778203] alg: No test for adler32 (adler32-zlib) [ 2330.766455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2330.801214] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2339.426096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 2350.114965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.171570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2350.190979] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 2351.262565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2351.996495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.130@tcp [ 2352.351258] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.130@tcp [ 2352.375722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2352.450881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2352.540071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2352.597281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2352.649942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2352.706951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2352.785324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2352.860315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2352.895361] LNet: 124588:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2352.911073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2352.953529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2353.007741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2353.053349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2353.140435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2353.194308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2354.114728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2354.383713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2354.442149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2354.529612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2354.599622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2354.685607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2354.755794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2354.839658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2354.885758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2354.916294] LNet: 124705:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2354.938784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2354.975453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2355.028873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2355.086357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2355.134687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2355.201722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2356.036569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.130@tcp [ 2356.382434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2356.418633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2356.459580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2356.491246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2356.532567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2356.574430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2356.614271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2356.675624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2356.714929] LNet: 124822:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2356.767740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2356.858157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2356.940313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2356.987713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2357.031782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2357.090304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2357.906947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2358.222210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2358.275318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2358.350193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2358.424722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2358.463854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2358.511892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2358.549490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2358.583278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2358.614698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2358.645945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2358.676829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2358.721236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2358.756854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2358.787111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2363.118514] LNet: 125024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2363.138771] LNetError: 125024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2363.152923] LNet: Removed LNI 192.168.206.30@tcp [ 2363.179251] LNet: Removed LNI 192.168.206.30@tcp1 [ 2363.740642] Key type .llcrypt unregistered [ 2363.745164] Key type ._llcrypt unregistered [ 2384.710558] Key type ._llcrypt registered [ 2384.712557] Key type .llcrypt registered [ 2386.018219] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 08:21:03 (1789129263) [ 2386.639534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2386.649895] alg: No test for adler32 (adler32-zlib) [ 2387.594229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2387.653961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2387.686850] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2388.446596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2388.469734] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2389.315294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2389.393461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2390.436058] LNet: Removed LNI 192.168.206.30@tcp [ 2398.429763] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 08:21:16 (1789129276) [ 2400.189768] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2402.214624] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 08:21:19 (1789129279) [ 2409.222628] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 08:21:27 (1789129287) [ 2416.913257] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 08:21:34 (1789129294) [ 2424.413123] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 08:21:42 (1789129302) [ 2432.251631] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 08:21:50 (1789129310) [ 2440.065740] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 08:21:57 (1789129317) [ 2448.055196] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 08:22:05 (1789129325) [ 2455.378323] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 08:22:13 (1789129333) [ 2462.124733] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 08:22:20 (1789129340) [ 2468.858809] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 08:22:26 (1789129346) [ 2474.934024] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 08:22:33 (1789129353) [ 2481.192983] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 08:22:39 (1789129359) [ 2487.917701] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 08:22:45 (1789129365) [ 2494.053470] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 08:22:51 (1789129371) [ 2501.690978] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 08:22:59 (1789129379) [ 2508.939481] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 08:23:06 (1789129386) [ 2515.387068] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 08:23:13 (1789129393) [ 2521.612341] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 08:23:19 (1789129399) [ 2521.694802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2521.716149] LNet: 136716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2521.721856] LNetError: 136716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2521.738341] LNet: Removed LNI 192.168.206.30@tcp2 [ 2521.770728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2521.917358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2521.950121] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2523.241650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2523.277766] LNet: 136864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2523.289271] LNetError: 136864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2523.324195] LNet: Removed LNI 192.168.206.30@tcp [ 2523.364796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2523.447664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2523.476017] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2529.627647] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 08:23:27 (1789129407) [ 2536.152703] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 08:23:34 (1789129414) [ 2542.761598] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 08:23:40 (1789129420) [ 2550.192508] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 08:23:48 (1789129428) [ 2557.211475] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 08:23:55 (1789129435) [ 2563.988431] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 08:24:01 (1789129441) [ 2571.051671] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 08:24:09 (1789129449) [ 2577.259584] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 08:24:15 (1789129455) [ 2583.407436] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 08:24:21 (1789129461) [ 2589.504509] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 08:24:27 (1789129467) [ 2595.655693] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 08:24:33 (1789129473) [ 2602.327521] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 08:24:40 (1789129480) [ 2609.267988] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2390 sec ========== 08:24:46 (1789129486) [ 2609.571744] LNet: 144328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2609.583199] LNetError: 144328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2609.612042] LNet: Removed LNI 192.168.206.30@tcp [ 2610.235797] Key type .llcrypt unregistered [ 2610.243819] Key type ._llcrypt unregistered [ 2622.704367] Key type ._llcrypt registered [ 2622.725891] Key type .llcrypt registered [ 2623.052524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2623.066550] alg: No test for adler32 (adler32-zlib) [ 2624.334589] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 2624.614345] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2626.303214] Key type lgssc registered [ 2627.424600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2692.952503] Lustre: Mounted lustre-client - version 2.17.57_137_gfce00d1 [ 2698.097614] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2707.140804] Lustre: Unmounted lustre-client [ 2744.747777] Key type lgssc unregistered [ 2745.080730] LNet: 147290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2745.088397] LNetError: 147290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2745.106441] LNet: Removed LNI 192.168.206.30@tcp [ 2746.186171] Key type .llcrypt unregistered [ 2746.188750] Key type ._llcrypt unregistered