[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 355600577 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003394] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007037] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008020] pid_max: default: 32768 minimum: 301 [ 0.009179] LSM: Security Framework initializing [ 0.010060] Yama: becoming mindful. [ 0.011000] SELinux: Initializing. [ 0.011073] *** VALIDATE selinux *** [ 0.018648] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022319] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025157] *** VALIDATE tmpfs *** [ 0.027352] *** VALIDATE proc *** [ 0.028204] *** VALIDATE cgroup *** [ 0.029011] *** VALIDATE cgroup2 *** [ 0.030266] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033022] Spectre V2 : User space: Vulnerable [ 0.034006] Speculative Store Bypass: Vulnerable [ 0.036856] debug: unmapping init [mem 0xffffffffb9659000-0xffffffffb9660fff] [ 0.038144] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039779] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040018] ... version: 2 [ 0.040959] ... bit width: 48 [ 0.041009] ... generic registers: 4 [ 0.041745] ... value mask: 0000ffffffffffff [ 0.042024] ... max period: 00007fffffffffff [ 0.043008] ... fixed-purpose events: 3 [ 0.043908] ... event mask: 000000070000000f [ 0.044211] rcu: Hierarchical SRCU implementation. [ 0.046099] smp: Bringing up secondary CPUs ... [ 0.047383] x86: Booting SMP configuration: [ 0.048013] .... node #0, CPUs: #1 #2 #3 [ 0.050590] smp: Brought up 1 node, 4 CPUs [ 0.051950] smpboot: Max logical packages: 1 [ 0.052010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.255030] node 0 deferred pages initialised in 202ms [ 0.259148] devtmpfs: initialized [ 0.260257] x86/mm: Memory block size: 128MB [ 0.262824] gcov: version magic: 0x41383552 [ 0.265380] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.269119] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.271365] pinctrl core: initialized pinctrl subsystem [ 0.273185] [ 0.273724] ************************************************************* [ 0.276013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.278011] ** ** [ 0.280015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.282017] ** ** [ 0.285011] ** This means that this kernel is built to expose internal ** [ 0.287011] ** IOMMU data structures, which may compromise security on ** [ 0.289014] ** your system. ** [ 0.291015] ** ** [ 0.293014] ** If you see this message and you are not debugging the ** [ 0.295028] ** kernel, report this immediately to your vendor! ** [ 0.298014] ** ** [ 0.300017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.302018] ************************************************************* [ 0.304910] NET: Registered protocol family 16 [ 0.306592] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.309082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.312078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.315139] cpuidle: using governor menu [ 0.316688] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.318851] PCI: Using configuration type 1 for base access [ 0.321158] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.330063] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.332027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.335176] cryptd: max_cpu_qlen set to 1000 [ 0.337257] ACPI: Added _OSI(Module Device) [ 0.339014] ACPI: Added _OSI(Processor Device) [ 0.341012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.342011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.347823] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.354061] ACPI: Interpreter enabled [ 0.355045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.356009] ACPI: Using IOAPIC for interrupt routing [ 0.358093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.361420] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.371436] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.373023] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.376031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.379070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.384404] acpiphp: Slot [2] registered [ 0.386098] acpiphp: Slot [5] registered [ 0.387085] acpiphp: Slot [6] registered [ 0.389128] acpiphp: Slot [3] registered [ 0.390075] acpiphp: Slot [4] registered [ 0.391061] acpiphp: Slot [7] registered [ 0.393055] acpiphp: Slot [8] registered [ 0.394070] acpiphp: Slot [9] registered [ 0.395061] acpiphp: Slot [10] registered [ 0.397062] acpiphp: Slot [11] registered [ 0.398055] acpiphp: Slot [12] registered [ 0.399060] acpiphp: Slot [13] registered [ 0.401063] acpiphp: Slot [14] registered [ 0.402088] acpiphp: Slot [15] registered [ 0.404073] acpiphp: Slot [16] registered [ 0.405059] acpiphp: Slot [17] registered [ 0.406057] acpiphp: Slot [18] registered [ 0.408058] acpiphp: Slot [19] registered [ 0.409059] acpiphp: Slot [20] registered [ 0.410057] acpiphp: Slot [21] registered [ 0.412081] acpiphp: Slot [22] registered [ 0.413048] acpiphp: Slot [23] registered [ 0.415079] acpiphp: Slot [24] registered [ 0.416058] acpiphp: Slot [25] registered [ 0.417043] acpiphp: Slot [26] registered [ 0.417864] acpiphp: Slot [27] registered [ 0.419040] acpiphp: Slot [28] registered [ 0.419876] acpiphp: Slot [29] registered [ 0.420038] acpiphp: Slot [30] registered [ 0.420905] acpiphp: Slot [31] registered [ 0.422046] PCI host bridge to bus 0000:00 [ 0.422991] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.425011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.427012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.429007] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.430006] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.432010] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.433137] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.434650] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.436836] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.442382] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.445427] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.447007] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.448006] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.450007] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.451459] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.453453] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.454017] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.456487] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.458006] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.464012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.467983] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.473719] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.479018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.484014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.499019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.506042] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.508653] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.511645] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.522015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.529506] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.531280] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.533161] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.534310] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.537181] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.539155] iommu: Default domain type: Passthrough [ 0.540321] SCSI subsystem initialized [ 0.541063] ACPI: bus type USB registered [ 0.541955] usbcore: registered new interface driver usbfs [ 0.543039] usbcore: registered new interface driver hub [ 0.544043] usbcore: registered new device driver usb [ 0.546150] pps_core: LinuxPPS API ver. 1 registered [ 0.547005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.549061] PTP clock support registered [ 0.551045] EDAC MC: Ver: 3.0.0 [ 0.553437] PCI: Using ACPI for IRQ routing [ 0.554807] NetLabel: Initializing [ 0.556009] NetLabel: domain hash size = 128 [ 0.557009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.559090] NetLabel: unlabeled traffic allowed by default [ 0.561148] vgaarb: loaded [ 0.562354] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.564010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.569744] clocksource: Switched to clocksource kvm-clock [ 0.718128] VFS: Disk quotas dquot_6.6.0 [ 0.719462] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.721662] *** VALIDATE ramfs *** [ 0.723036] *** VALIDATE hugetlbfs *** [ 0.724566] pnp: PnP ACPI init [ 0.726920] pnp: PnP ACPI: found 6 devices [ 0.741444] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.743308] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.744501] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.745967] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.747396] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.748743] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.751034] NET: Registered protocol family 2 [ 0.752848] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.756932] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.760157] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.763843] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.766310] TCP: Hash tables configured (established 65536 bind 65536) [ 0.768650] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.772899] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.776663] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.780449] NET: Registered protocol family 1 [ 0.782786] RPC: Registered named UNIX socket transport module. [ 0.784637] RPC: Registered udp transport module. [ 0.786280] RPC: Registered tcp transport module. [ 0.788117] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.789796] NET: Registered protocol family 44 [ 0.791729] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.794132] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.796514] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.799453] PCI: CLS 0 bytes, default 64 [ 0.801433] Unpacking initramfs... [ 2.159415] debug: unmapping init [mem 0xffff8eea7cc64000-0xffff8eea7ffcffff] [ 2.166610] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.168870] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.173523] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.657987] Initialise system trusted keyrings [ 2.659433] Key type blacklist registered [ 2.660952] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.669309] zbud: loaded [ 2.671591] *** VALIDATE nfs *** [ 2.672299] *** VALIDATE nfs4 *** [ 2.673322] pstore: using deflate compression [ 2.675393] Platform Keyring initialized [ 2.756618] NET: Registered protocol family 38 [ 2.758118] Key type asymmetric registered [ 2.759477] Asymmetric key parser 'x509' registered [ 2.761068] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.763541] io scheduler mq-deadline registered [ 2.765183] io scheduler kyber registered [ 2.767065] io scheduler bfq registered [ 2.768848] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.771455] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.773772] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.776095] ACPI: Power Button [PWRF] [ 2.780856] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.786539] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.794129] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.821760] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.849257] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.854197] Non-volatile memory driver v1.3 [ 2.855738] Linux agpgart interface v0.103 [ 2.887098] virtio_blk virtio1: [vda] 145872 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.889727] vda: detected capacity change from 0 to 74686464 [ 2.906897] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.913114] vdb: detected capacity change from 0 to 1073741824 [ 2.929065] libphy: Fixed MDIO Bus: probed [ 2.954284] usbcore: registered new interface driver usbserial_generic [ 2.961335] usbserial: USB Serial support registered for generic [ 2.983814] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.010343] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.013354] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.020673] mousedev: PS/2 mouse device common for all mice [ 3.029808] rtc_cmos 00:05: RTC can wake from S4 [ 3.034779] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.041076] rtc_cmos 00:05: registered as rtc0 [ 3.046889] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.053565] intel_pstate: CPU model not supported [ 3.061112] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.064759] hid: raw HID events driver (C) Jiri Kosina [ 3.074789] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.075852] usbcore: registered new interface driver usbhid [ 3.075856] usbhid: USB HID core driver [ 3.076136] drop_monitor: Initializing network drop monitor service [ 3.076275] Initializing XFRM netlink socket [ 3.076673] NET: Registered protocol family 10 [ 3.083948] Segment Routing with IPv6 [ 3.095493] NET: Registered protocol family 17 [ 3.097611] mpls_gso: MPLS GSO support [ 3.102943] RAS: Correctable Errors collector initialized. [ 3.105438] AVX version of gcm_enc/dec engaged. [ 3.106736] AES CTR mode by8 optimization enabled [ 3.275358] sched_clock: Marking stable (3275004762, 0)->(4088657411, -813652649) [ 3.280837] registered taskstats version 1 [ 3.284428] Loading compiled-in X.509 certificates [ 3.290358] zswap: loaded using pool lzo/zbud [ 3.346492] Key type big_key registered [ 3.394233] Key type encrypted registered [ 3.398324] ima: No TPM chip found, activating TPM-bypass! [ 3.402930] ima: Allocated hash algorithm: sha1 [ 3.406245] ima: No architecture policies found [ 3.409331] evm: Initialising EVM extended attributes: [ 3.411798] evm: security.selinux [ 3.413257] evm: security.ima [ 3.414552] evm: security.capability [ 3.415729] evm: HMAC attrs: 0x1 [ 3.418376] rtc_cmos 00:05: setting system clock to 2026-09-08 12:03:47 UTC (1788869027) [ 3.424569] debug: unmapping init [mem 0xffffffffba603000-0xffffffffba7fffff] [ 3.427951] debug: unmapping init [mem 0xffffffffb9382000-0xffffffffb9658fff] [ 3.436326] Write protecting the kernel read-only data: 28672k [ 3.441269] debug: unmapping init [mem 0xffffffffb7a03000-0xffffffffb7bfffff] [ 3.444543] debug: unmapping init [mem 0xffffffffb8314000-0xffffffffb83fffff] [ 3.483082] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.492379] systemd[1]: Detected virtualization kvm. [ 3.497126] systemd[1]: Detected architecture x86-64. [ 3.499296] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.531689] systemd[1]: No hostname configured. [ 3.534107] systemd[1]: Set hostname to . [ 3.536096] random: systemd: uninitialized urandom read (16 bytes read) [ 3.539168] systemd[1]: Initializing machine ID from random generator. [ 3.641043] random: ln: uninitialized urandom read (6 bytes read) [ 3.744974] random: systemd: uninitialized urandom read (16 bytes read) [ 3.752463] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.762030] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 3.768527] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.852879] device-mapper: uevent: version 1.0.3 [ 4.855257] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.012175] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 6.095235] scsi host0: ata_piix [ 6.149851] scsi host1: ata_piix [ 6.151364] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.153441] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.570261] random: fast init done [ 10.991275] random: crng init done [ 10.993477] random: 7 urandom warning(s) missed due to ratelimiting [ 12.262047] hrtimer: interrupt took 8545328 ns [ 12.459921] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.837665] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.243407] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.455223] SELinux: Disabled at runtime. [ 22.556358] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.578677] systemd[1]: Detected virtualization kvm. [ 22.580937] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.889535] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.904994] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.924605] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.946862] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.991822] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.025875] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.053175] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 25.729615] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 28.600908] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.731811] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.959160] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.958345] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 34.439146] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (11s / no limit) [** ] A start job is running for Configur…only root support (12s / no limit) [*** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ **] A start job is running for Configur…only root support (15s / no limit) [ *] A start job is running for Configur…only root support (15s / no limit) [ **] A start job is running for Configur…only root support (16s / no limit) [ ***[ 41.387655] Key type dns_resolver registered ] A start job is running for Configur…only root support (16s / no limit) [ *** ] A start job is running for Configur…only root support (17s / no limit)[ 42.527928] NFS: Registering the id_resolver key type [ 42.538448] Key type id_resolver registered [ 42.547539] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (17s / no limit) [*** ] A start job is running for Configur…only root support (18s / no limit) [** ] A start job is running for Configur…only root support (19s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg448-client login: [ 138.085363] libcfs: loading out-of-tree module taints kernel. [ 138.632755] Key type ._llcrypt registered [ 138.643259] Key type .llcrypt registered [ 139.601469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 139.650840] alg: No test for adler32 (adler32-zlib) [ 142.066670] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 143.596200] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 145.624268] Key type lgssc registered [ 149.135959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 295.853400] Lustre: Mounted lustre-client [ 300.552230] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 315.727410] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing check_logdir /tmp/testlogs/ [ 321.505210] Lustre: lustre-OST0000-osc-ffff8eeac9074800: disconnect after 23s idle [ 321.725390] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing yml_node [ 327.155774] Lustre: DEBUG MARKER: Client: 2.17.55.24 [ 329.926764] Lustre: DEBUG MARKER: MDS: 2.17.55.24 [ 332.938530] Lustre: DEBUG MARKER: OSS: 2.17.55.24 [ 334.520636] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 8 08:09:17 EDT 2026 [ 353.725231] Lustre: Unmounted lustre-client [ 386.450117] Key type lgssc unregistered [ 386.790803] LNet: 8403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.805020] LNetError: 8403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 386.841724] LNet: Removed LNI 192.168.204.48@tcp [ 387.968462] Key type .llcrypt unregistered [ 387.972836] Key type ._llcrypt unregistered [ 419.528951] Key type ._llcrypt registered [ 419.532339] Key type .llcrypt registered [ 419.832676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 419.863161] alg: No test for adler32 (adler32-zlib) [ 420.916624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 421.007848] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 421.021561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 425.867266] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 426.231305] LNet: 9289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.259451] LNetError: 9289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 426.298876] LNet: Removed LNI 192.168.204.48@tcp [ 427.009482] Key type .llcrypt unregistered [ 427.014464] Key type ._llcrypt unregistered [ 437.604487] Key type ._llcrypt registered [ 437.609113] Key type .llcrypt registered [ 439.490983] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 08:11:01 (1788869461) [ 439.973827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 440.004344] alg: No test for adler32 (adler32-zlib) [ 440.937708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.037785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.074626] LNet: 9703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.115405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.194603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 441.222927] LNet: 9710:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 449.631300] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 08:11:12 (1788869472) [ 449.755401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.803413] LNet: 10286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 449.850111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.977368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 458.208199] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 08:11:20 (1788869480) [ 458.309722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.347903] LNet: 10869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.364059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.424564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 466.289264] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 08:11:28 (1788869488) [ 466.593798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.622330] LNet: 11460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 466.643036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.731284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 473.696330] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 08:11:36 (1788869496) [ 473.803083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.831747] LNet: 12038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 473.853195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.997338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 474.781558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 483.504619] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 08:11:45 (1788869505) [ 483.889817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.923826] LNet: 12636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 483.966731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.116090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 484.884443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 485.810551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 494.149328] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 08:11:56 (1788869516) [ 494.345310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.425858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.552248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 502.269829] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 08:12:04 (1788869524) [ 502.575745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.613310] LNet: 13817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 502.620750] LNet: 13817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 502.657796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.760862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 502.805672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 503.274460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 503.312854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 503.918686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 503.986906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 504.437769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 504.480673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 505.165942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 505.249411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 506.012251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 506.081671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 506.611716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 506.677712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 507.467546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 507.526709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 508.154268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 508.215986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 508.806072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 508.862228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 509.480488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 509.531503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 510.292292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 510.353040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 510.982112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 511.063299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 520.214484] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 08:12:22 (1788869542) [ 520.439652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.523262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.632387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 520.695624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 529.115921] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 08:12:31 (1788869551) [ 529.190200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.311836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.518645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 529.593537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 536.686628] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 08:12:39 (1788869559) [ 536.882252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.909285] LNet: 15669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 536.918845] LNet: 15669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 536.943654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.042768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 537.077792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 545.445420] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 08:12:47 (1788869567) [ 545.630625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.772500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.928086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 546.011831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 554.040146] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 08:12:56 (1788869576) [ 554.227452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.295375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.409766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 554.454474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 562.996034] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 08:13:05 (1788869585) [ 563.092876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.168435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.318908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 563.382448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 572.077621] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 08:13:14 (1788869594) [ 572.193287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.254414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.322768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 572.909130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 579.610164] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 08:13:22 (1788869602) [ 579.863740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.921154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.142523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 580.379917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 589.414283] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 08:13:31 (1788869611) [ 589.519577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.612494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.774835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 590.210717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 598.807583] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 08:13:41 (1788869621) [ 598.957244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.038922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.763418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 600.033203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 607.672744] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 08:13:50 (1788869630) [ 608.016953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.036650] LNet: 20506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.042788] LNet: 20506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 608.069741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.151560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 608.250254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 616.309857] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 08:13:58 (1788869638) [ 616.439443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.489934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.590570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 616.675445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 624.150757] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 08:14:06 (1788869646) [ 624.502184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.554344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.647835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 631.189425] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 08:14:13 (1788869653) [ 631.322429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.395252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.536901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 639.154717] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 08:14:21 (1788869661) [ 639.356761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.437399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.561701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 640.154687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 647.978692] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 08:14:30 (1788869670) [ 648.116527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.182384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.315355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 648.363267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 655.923570] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 08:14:38 (1788869678) [ 656.043395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.117526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.254216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 656.300463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 663.765858] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 08:14:46 (1788869686) [ 663.873115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.946134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.140936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 664.209938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 671.961782] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 08:14:54 (1788869694) [ 672.164497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.243463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.345770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 672.400563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 678.956632] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 08:15:01 (1788869701) [ 679.056863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.142632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.268559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 679.345665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 679.374336] LNetError: 25823:0:(peer.c:2277:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 679.423035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 679.463742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 686.231240] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 08:15:08 (1788869708) [ 686.353137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.476442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.630636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BLcr5DMNM1 [ 686.790775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BLcr5DMNM1 [ 686.912261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BLcr5DMNM1 [ 694.631993] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 08:15:17 (1788869717) [ 694.711177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.765077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.825336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 694.883903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 694.943615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 701.390907] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 08:15:23 (1788869723) [ 701.633182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.716066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.797672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 701.821988] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 703.616485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 705.317163] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 707.336248] Key type lgssc registered [ 708.912185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 713.877616] LNetError: 27585:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.48@tcp99 has no available nets [ 717.208432] Key type lgssc unregistered [ 717.655230] LNetError: 28709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 717.704255] LNet: Removed LNI 192.168.204.48@tcp [ 718.413576] Key type .llcrypt unregistered [ 718.419268] Key type ._llcrypt unregistered [ 725.086321] Key type ._llcrypt registered [ 725.091667] Key type .llcrypt registered [ 726.355379] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 08:15:48 (1788869748) [ 727.033916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 727.051063] alg: No test for adler32 (adler32-zlib) [ 727.948953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.015501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 731.496490] LNet: Message forwarding will be enabled in 2s [ 734.561332] LNet: Message forwarding enabled [ 735.169191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 735.199821] LNet: Message forwarding will stop in 2s to 6s [ 739.868792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 740.879594] LNet: Message forwarding disabled [ 743.050396] LNet: Message forwarding will be enabled in 2s [ 747.043272] LNet: Message forwarding enabled [ 753.200587] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 08:16:15 (1788869775) [ 753.317047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.372063] LNet: 30113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.064183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.192800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 763.224438] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 08:16:25 (1788869785) [ 763.426776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.464151] LNet: 30699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.531597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.636766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 763.691606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 763.742982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 763.789311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 763.868661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 763.899913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 763.937561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 763.969327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 763.999877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 764.027138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 764.059137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 764.087925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 764.119652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 764.156551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 764.184360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 764.223495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 764.286567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 764.349664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 764.400844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 764.449914] LNetError: 30746:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 773.180759] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 08:16:35 (1788869795) [ 773.269598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 773.313114] LNet: 31310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.352564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.493355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 780.844165] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 08:16:43 (1788869803) [ 782.908493] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg448-server [ 784.757929] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 08:16:47 (1788869807) [ 784.907535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.945372] LNet: 32245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.970305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.081214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 785.108967] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 786.307933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[49-53]@tcp [ 786.349747] LNet: 32383:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 786.425547] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.49@tcp is being used as a gateway but routing feature is not turned on [ 786.446031] LNetError: 32254:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.49@tcp has gone from up to down [ 787.443451] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.52@tcp is being used as a gateway but routing feature is not turned on [ 787.479705] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 788.454071] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.49@tcp is being used as a gateway but routing feature is not turned on [ 788.488295] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 790.505135] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.49@tcp is being used as a gateway but routing feature is not turned on [ 790.521870] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 794.509569] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 08:16:56 (1788869816) [ 794.598054] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.49@tcp is being used as a gateway but routing feature is not turned on [ 794.623115] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 797.152622] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg448-server [ 799.500981] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 08:17:01 (1788869821) [ 799.611458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.638672] LNet: 33314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.663774] LNetError: 33314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.691492] LNet: Removed LNI 192.168.204.48@tcp [ 799.730636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.808936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.850829] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 800.711956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[49-54/2]@tcp [ 800.730367] LNet: 33437:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 800.739304] LNet: 33437:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 800.760876] LNetError: 33324:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.51@tcp has gone from up to down [ 800.768931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[49-54/2]@tcp [ 800.779833] LNetError: 33324:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 809.448100] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 08:17:11 (1788869831) [ 809.619033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.687658] LNet: 34025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.715020] LNetError: 34025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.762717] LNet: Removed LNI 192.168.204.48@tcp [ 809.820781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.986811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 810.064750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.138927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.254859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 810.366779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.465148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.521412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 810.596872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.629556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.682607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 810.792251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.827528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.876539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 810.956457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.997434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.113902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 811.309404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.369168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.452642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 811.551372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.592330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.688238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 811.817951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.871833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.971312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 812.120748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.149794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.213694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 812.313858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.367544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.410582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 812.461665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.503719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.584866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 821.346372] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 08:17:23 (1788869843) [ 824.357318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg448-server [ 827.027534] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 08:17:29 (1788869849) [ 830.224550] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg448-server [ 833.051771] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 08:17:35 (1788869855) [ 833.296740] LNet: 35430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.308975] LNet: 35430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 834.241994] Key type .llcrypt unregistered [ 834.246216] Key type ._llcrypt unregistered [ 842.542168] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 842.728977] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 842.741370] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 842.753945] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 845.969761] Key type ._llcrypt registered [ 845.974303] Key type .llcrypt registered [ 846.395495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 846.424648] alg: No test for adler32 (adler32-zlib) [ 847.474562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.569877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 847.619458] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 848.516600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 848.534565] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 851.395683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 851.417351] LNet: Removed LNI 10.1.2.3@tcp [ 851.571965] LNet: 36157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.580925] LNetError: 36157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.607743] LNet: Removed LNI 192.168.204.48@tcp [ 852.514136] Key type .llcrypt unregistered [ 852.518788] Key type ._llcrypt unregistered [ 867.317134] Key type ._llcrypt registered [ 867.319641] Key type .llcrypt registered [ 868.624472] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 08:18:11 (1788869891) [ 869.356507] Key type .llcrypt unregistered [ 869.365641] Key type ._llcrypt unregistered [ 875.946211] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 876.150935] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 876.155250] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 876.159880] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 878.858537] Key type ._llcrypt registered [ 878.862105] Key type .llcrypt registered [ 879.223506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 879.241663] alg: No test for adler32 (adler32-zlib) [ 880.305035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.396542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 880.432964] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 881.256943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 881.277348] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 884.314774] LNet: 37656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.324307] LNetError: 37656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 884.329304] LNet: Removed LNI 10.1.2.3@tcp [ 884.364208] LNet: Removed LNI 192.168.204.48@tcp [ 885.189760] Key type .llcrypt unregistered [ 885.194420] Key type ._llcrypt unregistered [ 899.599912] Key type ._llcrypt registered [ 899.607163] Key type .llcrypt registered [ 901.242766] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 08:18:43 (1788869923) [ 901.745596] Key type .llcrypt unregistered [ 901.747839] Key type ._llcrypt unregistered [ 909.505625] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 909.725539] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 909.734034] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 909.739503] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 912.447886] Key type ._llcrypt registered [ 912.451495] Key type .llcrypt registered [ 912.801643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 912.831782] alg: No test for adler32 (adler32-zlib) [ 913.778500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.888844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 913.935880] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 916.279152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 916.297540] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 918.159749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 918.180630] LNet: Removed LNI 10.1.2.3@tcp [ 918.199424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 918.251547] LNet: Removed LNI 10.1.2.31@tcp [ 918.269346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 918.293271] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.068484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 919.109264] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 920.219974] LNet: 39579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.229660] LNetError: 39579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 920.238461] LNet: Removed LNI 10.1.2.31@tcp [ 920.253128] LNet: Removed LNI 10.1.2.3@tcp [ 920.988661] Key type .llcrypt unregistered [ 920.992764] Key type ._llcrypt unregistered [ 933.376398] Key type ._llcrypt registered [ 933.378612] Key type .llcrypt registered [ 934.888785] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 08:19:17 (1788869957) [ 935.451815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 935.466459] alg: No test for adler32 (adler32-zlib) [ 936.424367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.526609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 936.545221] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 937.353396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 937.382289] LNet: 40532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.393960] LNetError: 40532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.417939] LNet: Removed LNI 192.168.204.48@tcp [ 937.440712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.493840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.48@tcp [ 937.510087] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 938.449529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.48@tcp [ 938.490355] LNet: Removed LNI 192.168.204.48@tcp [ 938.568780] LNet: 40668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.375272] Key type .llcrypt unregistered [ 939.382032] Key type ._llcrypt unregistered [ 953.470494] Key type ._llcrypt registered [ 953.479911] Key type .llcrypt registered [ 955.633271] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 08:19:37 (1788869977) [ 957.298881] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg448-server [ 958.972748] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 08:19:41 (1788869981) [ 959.559353] Key type .llcrypt unregistered [ 959.562777] Key type ._llcrypt unregistered [ 966.462570] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 966.571864] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 966.575239] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 966.581781] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 969.231984] Key type ._llcrypt registered [ 969.236551] Key type .llcrypt registered [ 969.444498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 969.468245] alg: No test for adler32 (adler32-zlib) [ 970.469644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.557116] LNet: Added LNI 192.168.204.48@tcp1 [8/256/0/180] [ 971.325758] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 973.755709] LNet: 42508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.762567] LNetError: 42508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.774599] LNet: Removed LNI 192.168.204.48@tcp1 [ 973.800684] LNet: Removed LNI 10.1.2.3@tcp2 [ 974.771432] Key type .llcrypt unregistered [ 974.778956] Key type ._llcrypt unregistered [ 988.212672] Key type ._llcrypt registered [ 988.214777] Key type .llcrypt registered [ 989.402778] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 08:20:12 (1788870012) [ 989.937790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 989.961975] alg: No test for adler32 (adler32-zlib) [ 990.942951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.017087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 991.039084] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 991.786760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 991.799764] LNet: 43478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.806752] LNetError: 43478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.818722] LNet: Removed LNI 192.168.204.48@tcp [ 991.838969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.883817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 991.906949] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 998.860920] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 08:20:21 (1788870021) [ 998.998904] LNet: 44060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.020773] LNetError: 44060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.056078] LNet: Removed LNI 192.168.204.48@tcp [ 999.810803] Key type .llcrypt unregistered [ 999.816427] Key type ._llcrypt unregistered [ 1007.796579] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1007.930123] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1007.932440] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1007.960422] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1010.711541] Key type ._llcrypt registered [ 1010.719161] Key type .llcrypt registered [ 1010.929512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1010.942836] alg: No test for adler32 (adler32-zlib) [ 1011.868488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.925118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1011.945692] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1012.835815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1012.862329] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1015.261424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1015.290782] LNet: 44813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.299025] LNetError: 44813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1015.308963] LNet: Removed LNI 10.1.2.3@tcp [ 1015.322420] LNet: Removed LNI 192.168.204.48@tcp [ 1015.348266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.457847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1015.509508] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1015.521258] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1016.341075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1016.377410] LNet: 44840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.391041] LNetError: 44840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1016.398370] LNetError: 44840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1016.414815] LNet: Removed LNI 10.1.2.3@tcp [ 1016.444556] LNet: Removed LNI 192.168.204.48@tcp [ 1016.508225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.602406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1016.647429] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1016.662911] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1017.412654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1017.438553] LNet: Added LNI 192.168.204.48@tcp2 [8/256/0/180] [ 1018.584919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1018.612544] LNet: 45002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.618116] LNetError: 45002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1018.624762] LNetError: 45002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1018.631791] LNet: Removed LNI 10.1.2.3@tcp [ 1018.635605] LNet: Removed LNI 192.168.204.48@tcp [ 1018.660445] LNet: Removed LNI 192.168.204.48@tcp2 [ 1018.704188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.776203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1018.808881] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1018.813981] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1018.819260] LNet: Added LNI 192.168.204.48@tcp2 [8/256/0/180] [ 1027.662978] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 08:20:50 (1788870050) [ 1027.810644] LNet: 45594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.820281] LNetError: 45594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1027.825912] LNetError: 45594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1027.832056] LNet: Removed LNI 10.1.2.3@tcp [ 1027.837594] LNet: Removed LNI 192.168.204.48@tcp [ 1027.861813] LNet: Removed LNI 192.168.204.48@tcp2 [ 1028.529120] Key type .llcrypt unregistered [ 1028.530909] Key type ._llcrypt unregistered [ 1036.122551] Key type ._llcrypt registered [ 1036.128710] Key type .llcrypt registered [ 1036.437971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1036.475411] alg: No test for adler32 (adler32-zlib) [ 1037.427092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.489405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1037.519308] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1038.618558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1038.651552] LNet: 45995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.655966] LNetError: 45995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.665778] LNet: Removed LNI 192.168.204.48@tcp [ 1038.692992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1038.787549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1038.847173] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1039.569559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1039.591351] LNet: 46025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1039.599552] LNetError: 46025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1039.612651] LNet: Removed LNI 192.168.204.48@tcp [ 1039.638874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1039.724703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1039.765197] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1040.495770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1040.539485] LNet: Removed LNI 192.168.204.48@tcp [ 1040.552574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1040.649896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1040.700050] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1041.350578] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1041.552385] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1041.564020] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1041.570221] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1043.956832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1044.008136] LNet: 46136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.021846] LNet: 46136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1044.041182] LNetError: 46136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.050885] LNetError: 46136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1044.090222] LNet: Removed LNI 192.168.204.48@tcp [ 1044.114519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1044.194238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1044.239411] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1051.181731] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 08:21:13 (1788870073) [ 1051.371488] LNet: 46729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.387975] LNetError: 46729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1051.413915] LNet: Removed LNI 192.168.204.48@tcp [ 1052.236828] Key type .llcrypt unregistered [ 1052.243757] Key type ._llcrypt unregistered [ 1059.294893] Key type ._llcrypt registered [ 1059.297829] Key type .llcrypt registered [ 1059.579750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1059.601761] alg: No test for adler32 (adler32-zlib) [ 1060.589964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.695133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1060.729089] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1062.001655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1062.042565] LNet: 47133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.054381] LNetError: 47133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.069876] LNet: Removed LNI 192.168.204.48@tcp [ 1062.089952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.191456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1062.234727] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1070.445949] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 08:21:32 (1788870092) [ 1070.545637] LNet: 47716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1070.556427] LNetError: 47716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1070.595043] LNet: Removed LNI 192.168.204.48@tcp [ 1071.346644] Key type .llcrypt unregistered [ 1071.351720] Key type ._llcrypt unregistered [ 1077.709590] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1077.949841] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1077.954624] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1077.960636] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1079.438354] Key type ._llcrypt registered [ 1079.442230] Key type .llcrypt registered [ 1079.706412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1079.731570] alg: No test for adler32 (adler32-zlib) [ 1080.677516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1080.739240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1080.766263] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1081.734519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1081.767564] LNet: 48168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.786138] LNetError: 48168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.804960] LNet: Removed LNI 192.168.204.48@tcp [ 1081.823553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.936882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1081.987516] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1089.610748] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 08:21:52 (1788870112) [ 1089.711528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1089.743678] LNet: 48761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.750621] LNetError: 48761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.778915] LNet: Removed LNI 192.168.204.48@tcp [ 1089.832493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1089.972673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1095.668777] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 08:21:58 (1788870118) [ 1095.838636] LNet: 49348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.455875] Key type .llcrypt unregistered [ 1096.461570] Key type ._llcrypt unregistered [ 1103.802278] Key type ._llcrypt registered [ 1103.806902] Key type .llcrypt registered [ 1104.066461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1104.099351] alg: No test for adler32 (adler32-zlib) [ 1105.104593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1105.177539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1105.201066] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1106.111807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1106.150443] LNet: 49748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.154900] LNetError: 49748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.167356] LNet: Removed LNI 192.168.204.48@tcp [ 1106.204292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1106.292940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1106.313644] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1114.584113] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 08:22:16 (1788870136) [ 1114.748401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1114.836805] LNet: 50334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.856849] LNetError: 50334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.939812] LNet: Removed LNI 192.168.204.48@tcp [ 1114.985434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1115.118318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1122.566755] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 08:22:24 (1788870144) [ 1124.148465] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1126.235403] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 08:22:28 (1788870148) [ 1127.935543] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1129.783946] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 08:22:32 (1788870152) [ 1131.413788] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1133.230862] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 08:22:35 (1788870155) [ 1134.770948] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1137.082916] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 08:22:39 (1788870159) [ 1139.293753] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1141.345032] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 08:22:43 (1788870163) [ 1141.571375] LNet: 52394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1142.480616] Key type .llcrypt unregistered [ 1142.486407] Key type ._llcrypt unregistered [ 1148.977862] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1149.188742] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1149.192088] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1149.205268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1151.937789] Key type ._llcrypt registered [ 1151.941433] Key type .llcrypt registered [ 1152.282632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1152.323947] alg: No test for adler32 (adler32-zlib) [ 1153.196176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1153.295130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1153.317789] LNet: Added LNI 192.168.204.48@tcp [8/512/0/180] [ 1153.329194] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1153.335175] LNet: 52720:0:(api-ni.c:2929:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1153.482080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1153.521952] LNet: 52739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1153.537434] LNetError: 52739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1153.547574] LNet: Removed LNI 10.1.2.3@tcp [ 1153.556723] LNetError: 52739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1153.588431] LNet: Removed LNI 192.168.204.48@tcp [ 1153.620207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1153.711150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1153.756886] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1153.875520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1153.911097] LNet: 52763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1153.928243] LNetError: 52763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1153.951764] LNet: Removed LNI 192.168.204.48@tcp [ 1154.027525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1154.150857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1154.191587] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1154.197734] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1154.202221] LNet: 52771:0:(api-ni.c:2929:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1154.356926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1154.399682] LNet: 52789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.415797] LNetError: 52789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1154.423222] LNet: Removed LNI 10.1.2.3@tcp [ 1154.425488] LNetError: 52789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1154.475558] LNet: Removed LNI 192.168.204.48@tcp [ 1154.546710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1154.750366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1154.815588] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1154.845701] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1154.847789] LNet: 52797:0:(api-ni.c:2929:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1155.181281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1155.231725] LNet: 52823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.258465] LNetError: 52823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1155.267786] LNet: Removed LNI 10.1.2.3@tcp [ 1155.276296] LNetError: 52823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.307119] LNet: Removed LNI 192.168.204.48@tcp [ 1155.346773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1155.435343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1155.465860] LNet: Added LNI 192.168.204.48@tcp [8/256/0/182] [ 1155.472491] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1155.627866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1155.651790] LNet: 52851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.663634] LNetError: 52851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1155.676550] LNet: Removed LNI 10.1.2.3@tcp [ 1155.679460] LNetError: 52851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.709294] LNet: Removed LNI 192.168.204.48@tcp [ 1155.747729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1155.848569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1155.878840] LNet: Added LNI 192.168.204.48@tcp [8/256/0/182] [ 1155.884155] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1155.886456] LNet: 52859:0:(api-ni.c:2929:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1156.029874] LNet: Removed LNI 10.1.2.3@tcp [ 1156.047746] LNet: Removed LNI 192.168.204.48@tcp [ 1156.755398] Key type .llcrypt unregistered [ 1156.756931] Key type ._llcrypt unregistered [ 1170.495467] Key type ._llcrypt registered [ 1170.498253] Key type .llcrypt registered [ 1172.166116] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 08:23:14 (1788870194) [ 1172.562835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1172.584547] alg: No test for adler32 (adler32-zlib) [ 1173.517900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1173.581896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1173.610750] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1174.647165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.48 [ 1174.689022] LNet: 53835:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1174.705707] LNet: Removed LNI 192.168.204.48@tcp [ 1183.399566] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 08:23:25 (1788870205) [ 1183.565740] LNet: 54397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.579838] LNetError: 54397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.612716] LNet: Removed LNI 192.168.204.48@tcp [ 1184.583606] Key type .llcrypt unregistered [ 1184.588167] Key type ._llcrypt unregistered [ 1192.252931] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1192.504940] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1194.165316] Key type ._llcrypt registered [ 1194.167850] Key type .llcrypt registered [ 1194.347425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1194.369698] alg: No test for adler32 (adler32-zlib) [ 1195.768470] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1201.946470] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 08:23:44 (1788870224) [ 1202.047649] LNet: 55301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.066135] LNetError: 55301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1202.095946] LNet: Removed LNI 10.1.2.3@tcp [ 1202.822747] Key type .llcrypt unregistered [ 1202.825582] Key type ._llcrypt unregistered [ 1209.626357] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1209.851892] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1209.855474] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1209.860051] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1212.548205] Key type ._llcrypt registered [ 1212.553879] Key type .llcrypt registered [ 1212.752499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1212.768066] alg: No test for adler32 (adler32-zlib) [ 1213.716803] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1220.125736] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 08:24:02 (1788870242) [ 1221.499597] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1223.438795] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 08:24:05 (1788870245) [ 1223.646780] LNet: 56493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.653825] LNetError: 56493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1223.693841] LNet: Removed LNI 10.1.2.3@tcp [ 1224.536631] Key type .llcrypt unregistered [ 1224.538290] Key type ._llcrypt unregistered [ 1231.372264] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1231.607339] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1231.617446] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1231.627384] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1234.423553] Key type ._llcrypt registered [ 1234.428211] Key type .llcrypt registered [ 1234.662562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1234.702457] alg: No test for adler32 (adler32-zlib) [ 1235.699854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1235.759972] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1244.560207] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 08:24:27 (1788870267) [ 1245.084882] LNet: 57723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1245.093507] LNetError: 57723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1245.115344] LNet: Removed LNI 10.1.2.3@tcp [ 1245.801507] Key type .llcrypt unregistered [ 1245.811314] Key type ._llcrypt unregistered [ 1251.487127] Key type ._llcrypt registered [ 1251.495641] Key type .llcrypt registered [ 1251.720288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1251.742280] alg: No test for adler32 (adler32-zlib) [ 1252.674492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1252.704214] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1259.516600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.148@tcp [ 1260.106883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_interrupt -r 1 [ 1260.139676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_interrupt -r 1 [ 1260.167323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1260.184295] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1260.383424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1260.418589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1260.448704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1260.495327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1260.535996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1260.585379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1260.654279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1260.702338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1260.730714] LNet: 58160:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1260.760438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1260.798051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1260.836778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1260.869985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1260.896277] LNet: 1 local NIs in recovery (showing 1): 192.168.204.48@tcp [ 1260.910861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1260.943070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1260.974576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1261.553718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_dropped -r 1 [ 1261.604893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_dropped -r 1 [ 1261.651822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1261.666985] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1261.891813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1261.911394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1261.932197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1261.953638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1261.973791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1261.995196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1262.023287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1262.069889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1262.102566] LNet: 58270:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1262.121986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1262.175058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1262.207533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1262.254940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1262.294596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1262.317503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1262.350926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1262.795425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_aborted -r 1 [ 1262.838311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_aborted -r 1 [ 1262.889966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1262.930239] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1262.937281] LNet: Skipped 1 previous similar message [ 1263.248869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1263.290642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1263.339124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1263.384638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1263.434094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1263.486806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1263.523690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1263.567598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1263.582198] LNet: 58379:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1263.604601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1263.634734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1263.673986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1263.730386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1263.768568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1263.790125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1263.816741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1264.223391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_no_route -r 1 [ 1264.277487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_no_route -r 1 [ 1264.305200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1264.501394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1264.542676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1264.602185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1264.640797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1264.670295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1264.701205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1264.733711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1264.765840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1264.792456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1264.818517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1264.855597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1264.904065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1264.939646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1265.014697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1265.059699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1265.591407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_timeout -r 1 [ 1265.638762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_timeout -r 1 [ 1265.669778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1265.684980] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1265.689788] LNet: Skipped 1 previous similar message [ 1265.893406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1265.921593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1265.950841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1265.976407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1266.001673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1266.025520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1266.046809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1266.067419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1266.079680] LNet: 58597:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1266.084577] LNet: 58597:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1266.107542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1266.138962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1266.174399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1266.218696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1266.266750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1266.316094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1266.348549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1266.865354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_error -r 1 [ 1266.908931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_error -r 1 [ 1266.941584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1267.140518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1267.177662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1267.226454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1267.266031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1267.311947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1267.348944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1267.392902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1267.425605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1267.463931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1267.499664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1267.534386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1267.570840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1267.620658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1267.651504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1267.682873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1267.957389] LNet: 58747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1267.966437] LNetError: 58747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.039313] LNet: Removed LNI 192.168.204.48@tcp [ 1269.403554] Key type .llcrypt unregistered [ 1269.412265] Key type ._llcrypt unregistered [ 1286.283459] Key type ._llcrypt registered [ 1286.285554] Key type .llcrypt registered [ 1287.678287] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 08:25:10 (1788870310) [ 1288.531615] Key type .llcrypt unregistered [ 1288.533496] Key type ._llcrypt unregistered [ 1294.866266] Key type ._llcrypt registered [ 1294.872827] Key type .llcrypt registered [ 1295.191415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1295.215920] alg: No test for adler32 (adler32-zlib) [ 1296.231251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1296.299138] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1304.328739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.148@tcp [ 1315.227599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1315.307887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1315.338498] LNet: Added LNI 192.168.204.48@tcp1 [8/256/0/180] [ 1316.596361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_interrupt -r 1 [ 1316.626121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e local_interrupt -r 1 [ 1316.665586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_interrupt -r 1 [ 1316.707953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e local_interrupt -r 1 [ 1316.767754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e local_interrupt -r 1 [ 1316.823877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e local_interrupt -r 1 [ 1316.882419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e local_interrupt -r 1 [ 1316.986369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e local_interrupt -r 1 [ 1317.035435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1317.067520] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1317.075540] LNet: 1 local NIs in recovery (showing 1): 192.168.204.48@tcp [ 1317.313689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1317.353922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1317.390949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1317.443096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1317.493657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1317.537700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1317.570584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1317.605719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1317.623358] LNet: 60214:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1317.640490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1317.690577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1317.727867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1317.759902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1317.786792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1317.824840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1317.850286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1318.462470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_dropped -r 1 [ 1318.522623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e local_dropped -r 1 [ 1318.613901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_dropped -r 1 [ 1318.700418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e local_dropped -r 1 [ 1318.791201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e local_dropped -r 1 [ 1318.875056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e local_dropped -r 1 [ 1318.933639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e local_dropped -r 1 [ 1318.964464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e local_dropped -r 1 [ 1319.017084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1319.037355] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1319.043490] LNet: Skipped 2 previous similar messages [ 1319.275390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1319.313475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1319.354661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1319.392767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1319.422815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1319.450316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1319.493766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1319.545448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1319.569750] LNet: 60335:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1319.616148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1319.672981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1319.700703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1319.727582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1319.762880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1319.791575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1319.830787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1320.327984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_aborted -r 1 [ 1320.394080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e local_aborted -r 1 [ 1320.446561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_aborted -r 1 [ 1320.494574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e local_aborted -r 1 [ 1320.526786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e local_aborted -r 1 [ 1320.564388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e local_aborted -r 1 [ 1320.609437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e local_aborted -r 1 [ 1320.637218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e local_aborted -r 1 [ 1320.663612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1320.689857] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1320.694458] LNet: Skipped 2 previous similar messages [ 1320.894664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1320.932237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1320.967215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1321.022127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1321.052458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1321.079516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1321.116433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1321.141773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1321.157354] LNet: 60457:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1321.172771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1321.204762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1321.262226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1321.299121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1321.334603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1321.365656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1321.398445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1322.123777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_no_route -r 1 [ 1322.171282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e local_no_route -r 1 [ 1322.215140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_no_route -r 1 [ 1322.282107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e local_no_route -r 1 [ 1322.343044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e local_no_route -r 1 [ 1322.401597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e local_no_route -r 1 [ 1322.460313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e local_no_route -r 1 [ 1322.540966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e local_no_route -r 1 [ 1322.621791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1322.897852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1322.945470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1322.982813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1323.014260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1323.050828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1323.096912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1323.143914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1323.201195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1323.231715] LNet: 60579:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1323.249603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1323.284312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1323.314733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1323.340546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1323.369874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1323.397229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1323.435556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1323.970588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_timeout -r 1 [ 1324.006573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e local_timeout -r 1 [ 1324.055950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_timeout -r 1 [ 1324.107222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e local_timeout -r 1 [ 1324.153536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e local_timeout -r 1 [ 1324.186983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e local_timeout -r 1 [ 1324.252451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e local_timeout -r 1 [ 1324.301601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e local_timeout -r 1 [ 1324.347050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1324.369409] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1324.377852] LNet: Skipped 5 previous similar messages [ 1324.619794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1324.649867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1324.676889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1324.703091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1324.733692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1324.781272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1324.842862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1324.894440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1324.943813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1324.981299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1325.012979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1325.040329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1325.075350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1325.108488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1325.148649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1325.628642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_error -r 1 [ 1325.658774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e local_error -r 1 [ 1325.710936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_error -r 1 [ 1325.748940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e local_error -r 1 [ 1325.778763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e local_error -r 1 [ 1325.817882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e local_error -r 1 [ 1325.874358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e local_error -r 1 [ 1325.920278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e local_error -r 1 [ 1325.953796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1326.236780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1326.285965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1326.335283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1326.371197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1326.395979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1326.423729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1326.461991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1326.489706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1326.521476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1326.548743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1326.585553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1326.619367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1326.647776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1326.676751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1326.706268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1331.596710] LNet: 60897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1331.609542] LNetError: 60897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1331.626115] LNet: Removed LNI 192.168.204.48@tcp [ 1332.648275] LNet: Removed LNI 192.168.204.48@tcp1 [ 1333.211715] Key type .llcrypt unregistered [ 1333.213823] Key type ._llcrypt unregistered [ 1350.291925] Key type ._llcrypt registered [ 1350.298167] Key type .llcrypt registered [ 1351.803787] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 08:26:14 (1788870374) [ 1352.654621] Key type .llcrypt unregistered [ 1352.657863] Key type ._llcrypt unregistered [ 1358.190635] Key type ._llcrypt registered [ 1358.192656] Key type .llcrypt registered [ 1358.377911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1358.392295] alg: No test for adler32 (adler32-zlib) [ 1359.329281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1359.372065] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1366.539233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.148@tcp [ 1367.262985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e remote_dropped -r 1 [ 1367.296685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e remote_dropped -r 1 [ 1367.341818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1367.361122] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1367.631191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1367.675330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1367.715243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1367.772271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1367.812708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1367.852535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1367.903887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1367.943938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1367.970584] LNet: 62154:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1368.004875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1368.042708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1368.075819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1368.119775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1368.163363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1368.217177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1368.251044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1369.002494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e remote_error -r 1 [ 1369.051276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e remote_error -r 1 [ 1369.107623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1369.136870] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1369.446723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1369.490427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1369.560188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1369.601555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1369.664797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1369.711751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1369.750163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1369.796476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1369.815532] LNet: 62263:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1369.834549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1369.859918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1369.887769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1369.926046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1369.974556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1370.032827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1370.073215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1370.585252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e remote_timeout -r 1 [ 1370.619502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e remote_timeout -r 1 [ 1370.658749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1370.702221] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1370.994753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1371.056717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1371.103702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1371.156733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1371.213344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1371.270479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1371.318640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1371.359513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1371.381900] LNet: 62373:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1371.405843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1371.445715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1371.488382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1371.535484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1371.577923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1371.626189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1371.663326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1371.996588] LNet: 62414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.006081] LNetError: 62414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.039289] LNet: Removed LNI 192.168.204.48@tcp [ 1372.557668] Key type .llcrypt unregistered [ 1372.562245] Key type ._llcrypt unregistered [ 1392.202257] Key type ._llcrypt registered [ 1392.205379] Key type .llcrypt registered [ 1393.868669] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 08:26:56 (1788870416) [ 1394.929596] Key type .llcrypt unregistered [ 1394.939377] Key type ._llcrypt unregistered [ 1402.594548] Key type ._llcrypt registered [ 1402.605482] Key type .llcrypt registered [ 1402.878742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1402.895936] alg: No test for adler32 (adler32-zlib) [ 1403.953922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1403.997811] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1414.628809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.148@tcp [ 1426.093971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1426.155296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1426.182342] LNet: Added LNI 192.168.204.48@tcp1 [8/256/0/180] [ 1427.880556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e remote_dropped -r 1 [ 1427.931039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e remote_dropped -r 1 [ 1428.023937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e remote_dropped -r 1 [ 1428.079875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e remote_dropped -r 1 [ 1428.118413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e remote_dropped -r 1 [ 1428.162434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e remote_dropped -r 1 [ 1428.204132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e remote_dropped -r 1 [ 1428.248687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e remote_dropped -r 1 [ 1428.296471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1428.322634] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1428.328778] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.148@tcp [ 1428.629648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1428.674754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1428.714926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1428.758334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1428.814197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1428.855382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1428.920283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1428.966973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1428.984231] LNet: 63881:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1429.015908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1429.060718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1429.102425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1429.142518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1429.187617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1429.232371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1429.271436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1429.307603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1429.996638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e remote_error -r 1 [ 1430.046878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e remote_error -r 1 [ 1430.099726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e remote_error -r 1 [ 1430.172405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e remote_error -r 1 [ 1430.243939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e remote_error -r 1 [ 1430.292528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e remote_error -r 1 [ 1430.358843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e remote_error -r 1 [ 1430.421372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e remote_error -r 1 [ 1430.504762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1430.532223] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1430.546498] LNet: Skipped 2 previous similar messages [ 1430.840939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1430.876935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1430.905322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1430.935677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1430.972200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1431.008497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1431.066432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1431.112464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1431.131825] LNet: 64004:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1431.151976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1431.186250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1431.219522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1431.274155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1431.330393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1431.375278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1431.425207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1431.481357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1432.181949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e remote_timeout -r 1 [ 1432.232843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e remote_timeout -r 1 [ 1432.281847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e remote_timeout -r 1 [ 1432.328941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e remote_timeout -r 1 [ 1432.404075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e remote_timeout -r 1 [ 1432.446332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e remote_timeout -r 1 [ 1432.490848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e remote_timeout -r 1 [ 1432.528496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e remote_timeout -r 1 [ 1432.569902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.148@tcp [ 1432.593750] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1432.861925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1432.897576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1432.951219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1433.002442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1433.031726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1433.079575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1433.132933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1433.176895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1433.199991] LNet: 64127:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1433.232130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1433.276701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1433.347274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1433.397872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1433.463479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1433.522367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1433.598552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1433.661280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1439.408995] LNet: 64205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1439.415980] LNetError: 64205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.424185] LNet: Removed LNI 192.168.204.48@tcp [ 1439.432930] LNet: Removed LNI 192.168.204.48@tcp1 [ 1439.848250] Key type .llcrypt unregistered [ 1439.850255] Key type ._llcrypt unregistered [ 1460.626933] Key type ._llcrypt registered [ 1460.629254] Key type .llcrypt registered [ 1461.976331] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 08:28:04 (1788870484) [ 1464.160829] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1466.186888] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 08:28:08 (1788870488) [ 1467.102824] Key type .llcrypt unregistered [ 1467.106470] Key type ._llcrypt unregistered [ 1474.585646] Key type ._llcrypt registered [ 1474.588142] Key type .llcrypt registered [ 1474.788916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1474.819554] alg: No test for adler32 (adler32-zlib) [ 1475.801459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1475.826556] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1484.148946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.148@tcp [ 1484.890382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e network_timeout -r 1 [ 1484.937189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e network_timeout -r 1 [ 1485.001176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.148@tcp [ 1485.032597] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1485.088468] LNet: 1 local NIs in recovery (showing 1): 192.168.204.48@tcp [ 1485.275590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1485.307597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1485.333217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1485.363907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1485.392597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1485.422601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1485.453045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1485.478945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1485.492968] LNet: 65753:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1485.512825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1485.537211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1485.567381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1485.594447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1485.621450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1485.653400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1485.853434] LNet: 65792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.862235] LNetError: 65792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.888912] LNet: Removed LNI 192.168.204.48@tcp [ 1487.251327] Key type .llcrypt unregistered [ 1487.257516] Key type ._llcrypt unregistered [ 1506.222320] Key type ._llcrypt registered [ 1506.224889] Key type .llcrypt registered [ 1506.407393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1506.417984] alg: No test for adler32 (adler32-zlib) [ 1507.356758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1507.390995] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1514.797554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.148@tcp [ 1523.529894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1523.587907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1523.616505] LNet: Added LNI 192.168.204.48@tcp1 [8/256/0/180] [ 1525.172831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e network_timeout -r 1 [ 1525.214141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e network_timeout -r 1 [ 1525.270119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e network_timeout -r 1 [ 1525.318785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e network_timeout -r 1 [ 1525.367288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e network_timeout -r 1 [ 1525.422705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e network_timeout -r 1 [ 1525.472068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e network_timeout -r 1 [ 1525.542776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e network_timeout -r 1 [ 1525.618184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.148@tcp [ 1525.656511] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1525.860615] LNet: 1 local NIs in recovery (showing 1): 192.168.204.48@tcp [ 1525.864392] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.148@tcp [ 1525.957451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1525.989502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1526.027843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1526.070865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1526.110295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1526.150291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1526.207486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1526.274718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1526.304933] LNet: 66670:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1526.323776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1526.370848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1526.407765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1526.454193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1526.484971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1526.543546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1531.471239] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1531.908504] LNet: 66744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1531.914597] LNetError: 66744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1531.921585] LNet: Removed LNI 192.168.204.48@tcp [ 1531.949180] LNet: Removed LNI 192.168.204.48@tcp1 [ 1532.426339] Key type .llcrypt unregistered [ 1532.430648] Key type ._llcrypt unregistered [ 1550.441643] Key type ._llcrypt registered [ 1550.444544] Key type .llcrypt registered [ 1552.019282] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 08:29:34 (1788870574) [ 1552.444158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1552.462951] alg: No test for adler32 (adler32-zlib) [ 1553.444217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1553.514439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1553.564706] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1554.411537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1554.427779] LNet: Added LNI 192.168.204.48@tcp1 [8/256/0/180] [ 1555.160549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 1555.233455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1555.284558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1555.322746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1555.352569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1555.379350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1555.405446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.48@tcp [ 1555.432636] LNetError: 67844:0:(api-ni.c:4416:lnet_ni_set_healthv()) manually adding local NI 192.168.204.48@tcp to recovery [ 1555.555394] LNet: 1 local NIs in recovery (showing 1): 192.168.204.48@tcp [ 1555.556598] LNet: There was an unexpected network error while writing to 192.168.204.48: rc = -22 [ 1557.600927] LNet: There was an unexpected network error while writing to 192.168.204.48: rc = -22 [ 1557.951940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1561.697706] LNet: There was an unexpected network error while writing to 192.168.204.48: rc = -22 [ 1562.834447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1562.887570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1562.967614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.003492] LNet: 67951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.016225] LNetError: 67951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1563.030191] LNet: Removed LNI 192.168.204.48@tcp [ 1563.060887] LNet: Removed LNI 192.168.204.48@tcp1 [ 1563.094705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.166474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1563.190452] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1564.136430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1564.165414] LNet: Added LNI 192.168.204.48@tcp1 [8/256/0/180] [ 1565.362423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 1565.390613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1565.453911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1565.490643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1565.530876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1565.567584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1565.603863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1565.645995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.48@tcp [ 1565.690258] LNetError: 68223:0:(api-ni.c:4416:lnet_ni_set_healthv()) manually adding local NI 192.168.204.48@tcp to recovery [ 1566.179148] LNet: There was an unexpected network error while writing to 192.168.204.48: rc = -22 [ 1569.413882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1572.322369] LNet: There was an unexpected network error while writing to 192.168.204.48: rc = -22 [ 1572.334513] LNet: Skipped 1 previous similar message [ 1577.636979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1577.684484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1577.717917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1585.963411] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 08:30:07 (1788870607) [ 1586.077284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.112281] LNet: 68933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.118474] LNetError: 68933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.123102] LNetError: 68933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1586.133334] LNet: Removed LNI 192.168.204.48@tcp [ 1586.145884] LNet: Removed LNI 192.168.204.48@tcp1 [ 1586.183408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.278848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1586.303332] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1587.564779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1587.606613] LNet: Added LNI 192.168.204.48@tcp1 [8/256/0/180] [ 1589.117851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 1589.236711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1589.288788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1589.368916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1589.455264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1589.536588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1589.601688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.48@tcp --health 0 [ 1589.838513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1590.304466] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.48@tcp [ 1590.304613] LNet: There was an unexpected network error while writing to 192.168.204.48: rc = -22 [ 1590.322977] LNet: Skipped 1 previous similar message [ 1605.669620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1605.819155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1605.856898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1605.891917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.48@tcp --health 500 [ 1605.993749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1612.780983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1612.878857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.901211] LNet: 69433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.912262] LNetError: 69433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.924162] LNetError: 69433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1612.936160] LNet: Removed LNI 192.168.204.48@tcp [ 1612.950650] LNet: Removed LNI 192.168.204.48@tcp1 [ 1612.982456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.058945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1613.105509] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1613.886645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1613.906508] LNet: Added LNI 192.168.204.48@tcp1 [8/256/0/180] [ 1614.733208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 1614.778737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1614.842710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1614.891056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1614.937787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1614.994559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1615.039191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1615.109987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.48@tcp --health 0 [ 1616.099374] LNet: There was an unexpected network error while writing to 192.168.204.48: rc = -22 [ 1616.106470] LNet: Skipped 3 previous similar messages [ 1616.526637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1626.815966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1626.840432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1626.872933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1633.724514] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 08:30:56 (1788870656) [ 1634.188405] LNet: 70479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.195480] LNetError: 70479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.201097] LNetError: 70479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1635.232203] LNet: Removed LNI 192.168.204.48@tcp [ 1635.261710] LNet: Removed LNI 192.168.204.48@tcp1 [ 1636.055500] Key type .llcrypt unregistered [ 1636.060678] Key type ._llcrypt unregistered [ 1642.493675] Key type ._llcrypt registered [ 1642.497601] Key type .llcrypt registered [ 1642.716938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1642.740985] alg: No test for adler32 (adler32-zlib) [ 1643.662647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1643.692621] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1653.095149] Lustre: DEBUG MARKER: Initial discovery [ 1653.225125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.148@tcp [ 1655.467606] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1655.625991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1655.687803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.148@tcp [ 1655.704825] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1655.738062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1655.779215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.148@tcp -r 1 -m GET -l 3 [ 1655.831814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.148@tcp -r 1 -m GET -e local_error [ 1655.873462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.148@tcp -r 1 -m PUT -l 6 [ 1655.969681] LNet: 1 local NIs in recovery (showing 1): 192.168.204.48@tcp [ 1657.219652] Lustre: DEBUG MARKER: Force oleg448-client.virtnet to discover 192.168.204.148@tcp (in background) [ 1657.276366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.148@tcp [ 1658.383590] Lustre: DEBUG MARKER: Force oleg448-server to discover 192.168.204.48@tcp [ 1660.320537] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1660.457414] Lustre: DEBUG MARKER: Wait for 71233 [ 1661.935420] Lustre: DEBUG MARKER: Finished wait on 71233 [ 1662.070908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1662.095146] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1662.111699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1662.346304] LNet: 71691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.363568] LNetError: 71691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.380680] LNet: Removed LNI 192.168.204.48@tcp [ 1662.745496] Key type .llcrypt unregistered [ 1662.749545] Key type ._llcrypt unregistered [ 1676.974329] Key type ._llcrypt registered [ 1676.977358] Key type .llcrypt registered [ 1678.476125] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 08:31:40 (1788870700) [ 1679.296150] Key type .llcrypt unregistered [ 1679.298420] Key type ._llcrypt unregistered [ 1685.577754] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1685.670069] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1685.682695] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1685.696035] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1688.370624] Key type ._llcrypt registered [ 1688.372930] Key type .llcrypt registered [ 1688.527201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1688.545730] alg: No test for adler32 (adler32-zlib) [ 1689.534265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.616062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1689.644394] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1690.353818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1690.377127] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1699.447884] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 08:32:01 (1788870721) [ 1699.584051] LNet: 73759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.593264] LNetError: 73759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1699.611361] LNet: Removed LNI 10.1.2.3@tcp [ 1699.655560] LNet: Removed LNI 192.168.204.48@tcp [ 1700.563512] Key type .llcrypt unregistered [ 1700.567521] Key type ._llcrypt unregistered [ 1707.340861] Key type ._llcrypt registered [ 1707.342257] Key type .llcrypt registered [ 1707.556976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1707.574767] alg: No test for adler32 (adler32-zlib) [ 1708.526901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.579643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1708.602891] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1709.345755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1709.364589] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1710.283538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.48@tcp [ 1711.392587] LNetError: Unexpected error -22 connecting to 192.168.204.48@tcp at host 192.168.204.48:988 [ 1716.466289] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 08:32:19 (1788870739) [ 1716.632902] LNet: 74892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.643177] LNetError: 74892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1717.664472] LNet: Removed LNI 10.1.2.3@tcp [ 1717.668452] LNetError: 74892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1717.685205] LNet: Removed LNI 192.168.204.48@tcp [ 1718.282304] Key type .llcrypt unregistered [ 1718.285098] Key type ._llcrypt unregistered [ 1723.836231] Key type ._llcrypt registered [ 1723.844330] Key type .llcrypt registered [ 1724.085217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1724.099462] alg: No test for adler32 (adler32-zlib) [ 1725.122535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.244059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1725.271674] LNet: Added LNI 192.168.204.48@tcp1 [8/256/0/180] [ 1726.020127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1726.054197] LNet: Added LNI 192.168.204.48@tcp2 [8/256/0/180] [ 1727.081502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.48@tcp1 --nid 192.168.204.48@tcp2 [ 1731.940917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.48@tcp1 192.168.204.48@tcp2 [ 1731.968133] LNetError: 75735:0:(lib-move.c:2066:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.48@tcp1 to 192.168.204.48@tcp2 - Route table may be misconfigured [ 1731.995469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.48@tcp2 192.168.204.48@tcp1 [ 1739.388386] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 08:32:41 (1788870761) [ 1739.513571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.539124] LNet: 76300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.545307] LNetError: 76300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.555120] LNet: Removed LNI 192.168.204.48@tcp1 [ 1740.586484] LNet: Removed LNI 192.168.204.48@tcp2 [ 1740.662754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.743423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1740.762723] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1741.435579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1741.450750] LNet: Added LNI 192.168.204.48@tcp1 [8/256/0/180] [ 1742.159858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 1742.196592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.48@tcp 192.168.204.48@tcp [ 1742.230320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.48@tcp1 192.168.204.48@tcp1 [ 1742.275424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.48@tcp -d 192.168.204.48@tcp -e network_timeout [ 1742.315583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -e network_timeout [ 1742.343789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -e network_timeout [ 1742.377244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -e network_timeout [ 1742.408413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 1742.434237] LNet: There was an unexpected network error while writing to 192.168.204.48: rc = -22 [ 1742.453259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1742.542643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1742.641762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1742.753932] LNet: 1 local NIs in recovery (showing 1): 192.168.204.48@tcp [ 1748.400697] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 08:32:51 (1788870771) [ 1748.503937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.541283] LNet: 77168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.553250] LNetError: 77168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1748.559619] LNetError: 77168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1748.569716] LNet: Removed LNI 192.168.204.48@tcp [ 1749.608144] LNet: Removed LNI 192.168.204.48@tcp1 [ 1749.629269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.701901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1749.901423] LNet: 77206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1750.185844] Key type .llcrypt unregistered [ 1750.190394] Key type ._llcrypt unregistered [ 1758.442548] Key type ._llcrypt registered [ 1758.447040] Key type .llcrypt registered [ 1759.765839] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 08:33:02 (1788870782) [ 1760.137347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1760.151779] alg: No test for adler32 (adler32-zlib) [ 1761.032415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.164452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1763.490367] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 08:33:06 (1788870786) [ 1763.687884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.725450] LNet: 78303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.763271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.819204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1763.842981] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1764.417680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1764.433502] LNet: Added LNI 192.168.204.48@tcp1 [8/256/0/180] [ 1765.095143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 1765.117745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp1 [ 1765.147236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp1 [ 1770.559613] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 08:33:13 (1788870793) [ 1772.483567] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg448-server [ 1774.253426] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 08:33:16 (1788870796) [ 1775.708568] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg448-server [ 1777.230522] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 08:33:20 (1788870800) [ 1779.079743] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg448-server [ 1781.037587] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 08:33:23 (1788870803) [ 1782.604421] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg448-server [ 1784.422421] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 08:33:27 (1788870807) [ 1786.024877] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg448-server [ 1787.466649] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 08:33:30 (1788870810) [ 1789.576611] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg448-server [ 1791.330049] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 08:33:34 (1788870814) [ 1793.085323] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg448-server [ 1794.963199] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 08:33:37 (1788870817) [ 1796.779159] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg448-server [ 1798.239747] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 08:33:41 (1788870821) [ 1799.799486] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg448-server [ 1801.103547] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 08:33:44 (1788870824) [ 1801.236126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.261811] LNet: 82181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.269979] LNetError: 82181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.276524] LNet: Removed LNI 192.168.204.48@tcp [ 1802.345178] LNet: Removed LNI 192.168.204.48@tcp1 [ 1802.384087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.458433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1802.482438] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1803.161640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1803.235451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 1803.363982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.393409] LNet: 82328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.401874] LNetError: 82328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.407828] LNetError: 82328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1803.420895] LNet: Removed LNI 192.168.204.48@tcp [ 1803.452602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.509686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1803.534975] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1804.114676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 1804.298704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.366649] LNet: Removed LNI 192.168.204.48@tcp [ 1805.390947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.446156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1805.476903] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1806.126565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1806.218823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 1806.389334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.416436] LNet: 82621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.422067] LNet: 82621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1806.430851] LNetError: 82621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.438097] LNetError: 82621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1806.453328] LNet: Removed LNI 192.168.204.48@tcp [ 1806.475902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.533704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1806.562529] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1807.478564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 1807.660460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.746897] LNet: Removed LNI 192.168.204.48@tcp [ 1808.786577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.850331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1808.874419] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1809.468533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1809.556909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 1809.727707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1809.760865] LNetError: 82913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1809.766398] LNetError: 82913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1809.782864] LNet: Removed LNI 192.168.204.48@tcp [ 1809.801823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.841712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1809.870258] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1810.445651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 1810.614396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1810.632365] LNet: 83058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1810.636633] LNet: 83058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1811.689062] LNet: Removed LNI 192.168.204.48@tcp [ 1811.730841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.785950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1811.824200] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1812.374215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1812.450909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 1812.641264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.735850] LNet: Removed LNI 192.168.204.48@tcp [ 1813.754807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.809861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1813.834536] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1814.471647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 1815.806988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1815.834987] LNetError: 83360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1815.844612] LNetError: 83360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1815.870775] LNet: Removed LNI 192.168.204.48@tcp [ 1815.896627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1815.967369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1815.997737] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1816.729276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1816.808530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 1817.031855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1818.094597] LNet: Removed LNI 192.168.204.48@tcp [ 1818.120455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.164702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1818.196060] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1818.777762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 1819.031236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1819.064770] LNet: 83653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.071655] LNet: 83653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1820.134882] LNet: Removed LNI 192.168.204.48@tcp [ 1820.173022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.231329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1820.259455] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1820.917990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1821.018665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 1821.349726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1822.444382] LNet: Removed LNI 192.168.204.48@tcp [ 1822.476962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1822.569992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1822.607397] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1823.397622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 1823.707580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1824.812252] LNet: Removed LNI 192.168.204.48@tcp [ 1824.853181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.922628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1824.971585] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1825.682140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1825.756756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1825.792794] LNetError: 84078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.801215] LNetError: 84078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1825.834915] LNet: Removed LNI 192.168.204.48@tcp [ 1825.859925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.907517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1825.929736] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1826.620934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1826.713073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.745862] LNet: Removed LNI 192.168.204.48@tcp [ 1826.765842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.819438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1826.909234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.973051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.041801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1827.143528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.207815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.303602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1827.392241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.438487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.516617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1827.555891] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1828.348472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.369188] LNet: Removed LNI 192.168.204.48@tcp [ 1828.384543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1828.462391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1834.176541] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 08:34:17 (1788870857) [ 1834.297271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.332925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.403500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1834.433261] LNet: Added LNI 192.168.204.48@tcp231 [8/256/0/180] [ 1835.595772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1835.629855] LNet: 85104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.637919] LNet: 85104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1835.662836] LNet: Removed LNI 192.168.204.48@tcp231 [ 1835.693847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1835.779393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1835.814158] LNet: Added LNI 192.168.204.48@tcp231 [8/256/0/180] [ 1835.825255] LNet: 85112:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1836.301824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1836.336620] LNet: Removed LNI 192.168.204.48@tcp231 [ 1836.353114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1836.373877] LNet: Added LNI 192.168.204.48@tcp231 [8/256/0/180] [ 1837.579857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1837.611969] LNet: Removed LNI 192.168.204.48@tcp231 [ 1837.640030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1837.729465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1837.758519] LNet: Added LNI 192.168.204.48@tcp231 [8/256/0/0] [ 1837.764444] LNet: 85264:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1838.241794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1838.283778] LNet: Removed LNI 192.168.204.48@tcp231 [ 1838.303362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1838.331183] LNet: Added LNI 192.168.204.48@tcp231 [8/256/0/0] [ 1839.584783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1839.622818] LNet: Removed LNI 192.168.204.48@tcp231 [ 1839.656569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1839.725168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1839.754675] LNet: Added LNI 192.168.204.48@tcp231 [8/256/0/60] [ 1839.762144] LNet: 85416:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1840.179245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1840.210736] LNet: Removed LNI 192.168.204.48@tcp231 [ 1840.228477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1840.279913] LNet: Added LNI 192.168.204.48@tcp231 [8/256/0/60] [ 1841.445355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1841.475117] LNet: Removed LNI 192.168.204.48@tcp231 [ 1841.501348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.574243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1841.594271] LNet: Added LNI 192.168.204.48@tcp231 [8/256/0/180] [ 1841.996140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1842.030539] LNet: Removed LNI 192.168.204.48@tcp231 [ 1842.046535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1842.086774] LNet: Added LNI 192.168.204.48@tcp231 [8/256/0/180] [ 1848.474168] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 08:34:31 (1788870871) [ 1848.597147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1848.617481] LNetError: 86276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1848.621396] LNetError: 86276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1848.636098] LNet: Removed LNI 192.168.204.48@tcp231 [ 1848.651848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1848.708703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1848.730518] LNet: Added LNI 192.168.204.48@tcp232 [8/256/0/180] [ 1849.323231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1849.351845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1849.384118] LNet: Removed LNI 192.168.204.48@tcp232 [ 1849.407918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1849.437113] LNet: Added LNI 192.168.204.48@tcp232 [8/256/0/180] [ 1850.158537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1850.216946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1850.263249] LNet: Removed LNI 192.168.204.48@tcp232 [ 1850.279843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1850.304529] LNet: Added LNI 192.168.204.48@tcp232 [8/256/0/180] [ 1851.003754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1851.057978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1851.097227] LNet: Removed LNI 192.168.204.48@tcp232 [ 1851.120829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1851.144221] LNet: Added LNI 192.168.204.48@tcp232 [8/256/0/180] [ 1851.876568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1851.920540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1851.970038] LNet: Removed LNI 192.168.204.48@tcp232 [ 1851.991453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1858.258774] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 08:34:41 (1788870881) [ 1859.071544] Key type .llcrypt unregistered [ 1859.074916] Key type ._llcrypt unregistered [ 1864.347749] Key type ._llcrypt registered [ 1864.349838] Key type .llcrypt registered [ 1864.544890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1864.566323] alg: No test for adler32 (adler32-zlib) [ 1865.505394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1865.539322] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1872.442510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.148@tcp [ 1881.756698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.808455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1881.831315] LNet: Added LNI 192.168.204.48@tcp1 [8/256/0/180] [ 1883.206662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_interrupt -r 2 [ 1883.263089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e local_interrupt -r 2 [ 1883.313549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_interrupt -r 2 [ 1883.372548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e local_interrupt -r 2 [ 1883.429160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e local_interrupt -r 2 [ 1883.502846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e local_interrupt -r 2 [ 1883.552891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e local_interrupt -r 2 [ 1883.597796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e local_interrupt -r 2 [ 1883.640732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.48@tcp 192.168.204.148@tcp [ 1883.662546] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1883.667749] LNet: 1 local NIs in recovery (showing 1): 192.168.204.48@tcp [ 1883.933954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1883.966885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1884.000340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1884.043549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1884.108067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1884.189225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1884.240439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1884.291954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1884.329374] LNet: 88066:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1884.365093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1884.414202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1884.461940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1884.504488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1884.548938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1884.587681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1885.078146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_dropped -r 2 [ 1885.115095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e local_dropped -r 2 [ 1885.154411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_dropped -r 2 [ 1885.186980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e local_dropped -r 2 [ 1885.228126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e local_dropped -r 2 [ 1885.272728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e local_dropped -r 2 [ 1885.306436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e local_dropped -r 2 [ 1885.347498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e local_dropped -r 2 [ 1885.392384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.48@tcp 192.168.204.148@tcp [ 1885.420281] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1885.641545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1885.678365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1885.725770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1885.752951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1885.803923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1885.830737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1885.853382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1885.877509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1885.892744] LNet: 88186:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1885.918448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1885.945989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1885.985985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1886.012303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1886.039420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1886.068891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1886.541443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_aborted -r 2 [ 1886.577042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e local_aborted -r 2 [ 1886.613409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_aborted -r 2 [ 1886.650215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e local_aborted -r 2 [ 1886.692490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e local_aborted -r 2 [ 1886.745652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e local_aborted -r 2 [ 1886.797395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e local_aborted -r 2 [ 1886.846683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e local_aborted -r 2 [ 1886.888298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.48@tcp 192.168.204.148@tcp [ 1886.923467] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1887.152500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1887.182922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1887.219457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1887.248861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1887.285849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1887.320414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1887.359270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1887.401313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1887.415909] LNet: 88307:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1887.429764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1887.456483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1887.484257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1887.511298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1887.548801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1887.579524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1888.047546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_no_route -r 2 [ 1888.087282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e local_no_route -r 2 [ 1888.134317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_no_route -r 2 [ 1888.175940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e local_no_route -r 2 [ 1888.218781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e local_no_route -r 2 [ 1888.255773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e local_no_route -r 2 [ 1888.302322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e local_no_route -r 2 [ 1888.335407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e local_no_route -r 2 [ 1888.374083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.48@tcp 192.168.204.148@tcp [ 1888.624765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1888.659876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1888.694851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1888.732794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1888.769420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1888.805496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1888.849297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1888.891460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1888.934364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1888.962740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1888.988918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1889.019259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1889.054839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1889.091264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1889.626800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e local_timeout -r 2 [ 1889.673360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e local_timeout -r 2 [ 1889.715802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e local_timeout -r 2 [ 1889.768848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e local_timeout -r 2 [ 1889.812546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e local_timeout -r 2 [ 1889.864129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e local_timeout -r 2 [ 1889.904975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e local_timeout -r 2 [ 1889.961383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e local_timeout -r 2 [ 1890.006204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.48@tcp 192.168.204.148@tcp [ 1890.044180] LNet: There was an unexpected network error while writing to 192.168.204.148: rc = -22 [ 1890.050120] LNet: Skipped 1 previous similar message [ 1890.299215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1890.324914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1890.352246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1890.387800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1890.416849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1890.443631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1890.475885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1890.511419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1890.536293] LNet: 88547:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1890.542111] LNet: 88547:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1890.558388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1890.596452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1890.630458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1890.659390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1890.686508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1890.706302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1891.178325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp -m GET -e remote_dropped -r 2 [ 1891.254223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp -d 192.168.204.148@tcp1 -m GET -e remote_dropped -r 2 [ 1891.301970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp -m GET -e remote_dropped -r 2 [ 1891.338564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp -d 192.168.204.48@tcp1 -m GET -e remote_dropped -r 2 [ 1891.387337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp -m GET -e remote_dropped -r 2 [ 1891.432820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.48@tcp1 -d 192.168.204.148@tcp1 -m GET -e remote_dropped -r 2 [ 1891.491989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp -m GET -e remote_dropped -r 2 [ 1891.550335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.48@tcp1 -d 192.168.204.48@tcp1 -m GET -e remote_dropped -r 2 [ 1891.626379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.48@tcp 192.168.204.148@tcp [ 1891.657583] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.148@tcp [ 1891.979140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1892.011700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1892.042629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1892.079362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1892.109529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1892.151662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1892.215341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1892.256843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1892.317406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1892.369309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1892.419810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1892.473141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1892.501569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1892.536734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1898.226697] LNet: 88741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.235472] LNetError: 88741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1898.244053] LNet: Removed LNI 192.168.204.48@tcp [ 1898.256349] LNet: Removed LNI 192.168.204.48@tcp1 [ 1898.716880] Key type .llcrypt unregistered [ 1898.722949] Key type ._llcrypt unregistered [ 1919.384872] Key type ._llcrypt registered [ 1919.386925] Key type .llcrypt registered [ 1920.527888] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 08:35:43 (1788870943) [ 1920.927879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1920.942242] alg: No test for adler32 (adler32-zlib) [ 1921.865824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.939995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1921.968343] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1922.747235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1922.777816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.48@tcp [ 1922.797847] LNetError: 89721:0:(api-ni.c:4416:lnet_ni_set_healthv()) manually adding local NI 192.168.204.48@tcp to recovery [ 1922.913322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.48@tcp [ 1922.916437] LNet: 1 local NIs in recovery (showing 1): 192.168.204.48@tcp [ 1923.056093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1928.518961] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 08:35:51 (1788870951) [ 1928.618487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1928.641000] LNet: 90306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.648432] LNetError: 90306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.665246] LNet: Removed LNI 192.168.204.48@tcp [ 1928.700429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.760272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1928.782855] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1929.380728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1929.406792] LNet: Added LNI 192.168.204.48@tcp1 [8/256/0/180] [ 1930.341231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.48@tcp [ 1930.356982] LNetError: 90578:0:(api-ni.c:4416:lnet_ni_set_healthv()) manually adding local NI 192.168.204.48@tcp to recovery [ 1930.794630] LNet: There was an unexpected network error while writing to 192.168.204.48: rc = -22 [ 1931.576983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.48@tcp [ 1939.236771] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 08:36:02 (1788870962) [ 1940.922022] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg448-server [ 1942.514378] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 08:36:05 (1788870965) [ 1944.164747] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg448-server [ 1945.688252] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 08:36:08 (1788870968) [ 1947.391237] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg448-server [ 1948.802183] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 08:36:11 (1788870971) [ 1948.911025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1948.938078] LNet: 92202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.946077] LNetError: 92202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.950917] LNet: Removed LNI 192.168.204.48@tcp [ 1948.962340] LNet: Removed LNI 192.168.204.48@tcp1 [ 1948.993938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.054754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1949.077589] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1949.667750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1949.722896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1949.756988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1949.817505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1949.850799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1949.883666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1955.881868] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 08:36:18 (1788870978) [ 1956.142500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1956.192267] LNet: 92933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.214086] LNetError: 92933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1956.222495] LNetError: 92933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1956.249942] LNet: Removed LNI 192.168.204.48@tcp [ 1956.279724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1956.366798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1956.395414] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1963.434204] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 08:36:26 (1788870986) [ 1963.531060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.546060] LNet: 93631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.553125] LNetError: 93631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.574429] LNet: Removed LNI 192.168.204.48@tcp [ 1963.605541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.643908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1963.669484] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1969.649154] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 08:36:32 (1788870992) [ 1970.982205] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1972.520744] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 08:36:35 (1788870995) [ 1972.945991] LNet: 94671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.954262] LNetError: 94671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1972.969690] LNet: Removed LNI 192.168.204.48@tcp [ 1973.484714] Key type .llcrypt unregistered [ 1973.486961] Key type ._llcrypt unregistered [ 1978.642308] Key type ._llcrypt registered [ 1978.651850] Key type .llcrypt registered [ 1978.857372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1978.874312] alg: No test for adler32 (adler32-zlib) [ 1979.792429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1979.815865] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 1986.682542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.148@tcp [ 1991.255390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.148@tcp [ 1991.630745] LNet: 95088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.637060] LNetError: 95088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1991.653943] LNet: Removed LNI 192.168.204.48@tcp [ 1992.115446] Key type .llcrypt unregistered [ 1992.117227] Key type ._llcrypt unregistered [ 2001.766290] Key type ._llcrypt registered [ 2001.768313] Key type .llcrypt registered [ 2002.981875] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2004.662669] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2006.262984] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 08:37:09 (1788871029) [ 2007.680337] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2009.689404] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 08:37:12 (1788871032) [ 2011.376424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg448-server [ 2013.073065] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 08:37:15 (1788871035) [ 2015.132412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg448-server [ 2016.823338] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 08:37:19 (1788871039) [ 2018.231613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2018.264324] alg: No test for adler32 (adler32-zlib) [ 2019.416844] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 2019.720661] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2021.368292] Key type lgssc registered [ 2022.937636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2042.592801] Key type lgssc unregistered [ 2042.910583] LNet: 98261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2042.916110] LNetError: 98261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2042.931752] LNet: Removed LNI 192.168.204.48@tcp [ 2043.714631] Key type .llcrypt unregistered [ 2043.718938] Key type ._llcrypt unregistered [ 2059.289227] Key type ._llcrypt registered [ 2059.292714] Key type .llcrypt registered [ 2060.827879] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 08:38:03 (1788871083) [ 2061.261037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2061.289206] alg: No test for adler32 (adler32-zlib) [ 2062.220425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2062.296441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2062.339711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2062.397282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2062.476402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2068.685719] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 08:38:11 (1788871091) [ 2068.806457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2068.822753] LNet: 99672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.858226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2068.918265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2068.937304] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2069.666331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2069.683675] LNet: 99820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2069.689227] LNetError: 99820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2069.711384] LNet: Removed LNI 192.168.204.48@tcp [ 2069.727389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2069.766955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2069.786905] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2070.187521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2070.222227] LNetError: 99848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2070.264731] LNet: Removed LNI 192.168.204.48@tcp [ 2070.292984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2070.351582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2070.370737] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2076.123505] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 08:38:18 (1788871098) [ 2076.254309] LNet: 100433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.266492] LNet: 100433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2076.274502] LNetError: 100433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.285610] LNet: Removed LNI 192.168.204.48@tcp [ 2076.944546] Key type .llcrypt unregistered [ 2076.948363] Key type ._llcrypt unregistered [ 2083.355977] Key type ._llcrypt registered [ 2083.358538] Key type .llcrypt registered [ 2083.547100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2083.567812] alg: No test for adler32 (adler32-zlib) [ 2084.492833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.602836] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2084.607750] LNet: *** cfs_fail_loc=e003, val=100*** [ 2085.221361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 2085.235899] LNet: *** cfs_fail_loc=e003, val=100*** [ 2085.238963] LNet: Skipped 2 previous similar messages [ 2085.245049] Ping reply received with mapping: [ 2085.247273] found metadata reply NI[192.168.204.48@tcp] [ 2085.249512] 30 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2085.255185] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2085.257333] 5a 5a 5a 5a 5a 5a 5a 5a [ 2085.402939] LNet: 100835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.414570] LNetError: 100835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.430171] LNet: Removed LNI 192.168.204.48@tcp [ 2086.203750] Key type .llcrypt unregistered [ 2086.206195] Key type ._llcrypt unregistered [ 2096.446524] Key type ._llcrypt registered [ 2096.448557] Key type .llcrypt registered [ 2097.751679] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 08:38:40 (1788871120) [ 2098.162307] Key type .llcrypt unregistered [ 2098.164946] Key type ._llcrypt unregistered [ 2103.093974] Key type ._llcrypt registered [ 2103.097114] Key type .llcrypt registered [ 2103.254354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2103.266965] alg: No test for adler32 (adler32-zlib) [ 2104.311663] LNetError: 101881:0:(api-ni.c:2793:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2104.674391] Key type .llcrypt unregistered [ 2104.680675] Key type ._llcrypt unregistered [ 2109.177313] Key type ._llcrypt registered [ 2109.179571] Key type .llcrypt registered [ 2110.529823] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 08:38:53 (1788871133) [ 2111.179628] Key type .llcrypt unregistered [ 2111.183488] Key type ._llcrypt unregistered [ 2117.809735] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2117.813280] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2120.410510] Key type ._llcrypt registered [ 2120.412481] Key type .llcrypt registered [ 2120.579539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2120.600233] alg: No test for adler32 (adler32-zlib) [ 2121.512181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.583877] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2129.259401] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 08:39:12 (1788871152) [ 2129.407734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.429394] LNet: 103661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2129.436120] LNetError: 103661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2129.451804] LNet: Removed LNI 10.1.2.3@tcp [ 2129.475195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.036399] Key type .llcrypt unregistered [ 2130.039367] Key type ._llcrypt unregistered [ 2135.646796] Key type ._llcrypt registered [ 2135.650276] Key type .llcrypt registered [ 2135.801277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2135.823400] alg: No test for adler32 (adler32-zlib) [ 2136.760298] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2136.827551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 2136.948364] LNet: 103949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2136.953709] LNetError: 103949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2136.968220] LNet: Removed LNI 192.168.204.48@tcp [ 2137.694660] Key type .llcrypt unregistered [ 2137.698346] Key type ._llcrypt unregistered [ 2148.998872] Key type ._llcrypt registered [ 2149.001949] Key type .llcrypt registered [ 2150.605308] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 08:39:32 (1788871172) [ 2151.246487] Key type .llcrypt unregistered [ 2151.250525] Key type ._llcrypt unregistered [ 2157.798817] Key type ._llcrypt registered [ 2157.804209] Key type .llcrypt registered [ 2158.144646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2158.171229] alg: No test for adler32 (adler32-zlib) [ 2165.253388] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 08:39:47 (1788871187) [ 2165.411450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2165.473661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.619946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2165.671717] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2166.668913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2166.710626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2166.766562] LNet: Removed LNI 192.168.204.48@tcp [ 2167.172926] LNet: 105692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2167.633517] Key type .llcrypt unregistered [ 2167.636703] Key type ._llcrypt unregistered [ 2179.682484] Key type ._llcrypt registered [ 2179.688350] Key type .llcrypt registered [ 2181.181632] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 08:40:03 (1788871203) [ 2181.580786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2181.599425] alg: No test for adler32 (adler32-zlib) [ 2182.570838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.667304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2182.702355] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2183.484522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 2183.527090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.48@tcp [ 2183.548526] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2183.552858] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2183.556985] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2183, 0, 0 [ 2183.560127] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2183.563474] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2183.568269] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2183.572131] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2183, 0, 0 [ 2183.577959] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2183.580780] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2183.585666] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2183.589687] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2183, 0, 2233 [ 2183.599346] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2183.605899] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2183.610521] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2183.614102] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2183, 0, 2233 [ 2183.620046] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2190.464760] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 08:40:12 (1788871212) [ 2190.924306] LNet: 107232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2190.929089] LNetError: 107232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2191.983054] LNet: Removed LNI 192.168.204.48@tcp [ 2192.666956] Key type .llcrypt unregistered [ 2192.674624] Key type ._llcrypt unregistered [ 2198.444605] Key type ._llcrypt registered [ 2198.446900] Key type .llcrypt registered [ 2198.653662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2198.665614] alg: No test for adler32 (adler32-zlib) [ 2199.571512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2199.605525] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2206.938166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.148@tcp [ 2216.729459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.813357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2216.864762] LNet: Added LNI 192.168.204.48@tcp1 [8/256/0/180] [ 2217.899955] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2218.035205] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2218.041390] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2218.047184] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2220.434499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2220.453426] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2230.291955] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 08:40:53 (1788871253) [ 2230.387487] LNet: 108729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2230.411877] LNetError: 108729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2230.418359] LNet: Removed LNI 192.168.204.48@tcp [ 2230.424884] LNet: Removed LNI 192.168.204.48@tcp1 [ 2230.439467] LNet: Removed LNI 10.1.2.3@tcp99 [ 2231.116625] Key type .llcrypt unregistered [ 2231.121524] Key type ._llcrypt unregistered [ 2237.831765] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2237.843864] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2240.502352] Key type ._llcrypt registered [ 2240.508450] Key type .llcrypt registered [ 2240.785924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2240.815533] alg: No test for adler32 (adler32-zlib) [ 2241.834661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.944580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2241.990527] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2242.887403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2242.908453] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2245.289340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2245.347695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2245.401205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2245.432427] LNetError: 109483:0:(peer.c:2277:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2245.490103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2245.598600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2246.140805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2246.235914] LNet: 109503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2246.246326] LNetError: 109503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2246.253302] LNet: Removed LNI 10.1.2.3@tcp [ 2247.276605] LNet: Removed LNI 192.168.204.48@tcp [ 2248.379743] Key type .llcrypt unregistered [ 2248.382080] Key type ._llcrypt unregistered [ 2262.080324] Key type ._llcrypt registered [ 2262.082612] Key type .llcrypt registered [ 2263.352874] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 08:41:26 (1788871286) [ 2263.899819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2263.921513] alg: No test for adler32 (adler32-zlib) [ 2264.894799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2264.965205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2265.003354] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2265.774887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.48@tcp [ 2265.863624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg448-client@tcp [ 2272.763516] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 08:41:35 (1788871295) [ 2273.042148] LNet: 111038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.047455] LNetError: 111038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.069195] LNet: Removed LNI 192.168.204.48@tcp [ 2273.783758] Key type .llcrypt unregistered [ 2273.789140] Key type ._llcrypt unregistered [ 2280.240429] Key type ._llcrypt registered [ 2280.242779] Key type .llcrypt registered [ 2280.437265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2280.468119] alg: No test for adler32 (adler32-zlib) [ 2281.388743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2281.411923] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2281.444594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2287.663923] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 08:41:50 (1788871310) [ 2289.107832] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2290.726882] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 08:41:53 (1788871313) [ 2292.067471] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2293.758033] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 08:41:56 (1788871316) [ 2294.121516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2294.171924] LNet: 112457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.180961] LNetError: 112457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2294.193744] LNet: Removed LNI 192.168.204.48@tcp [ 2294.217764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2294.313971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2294.365573] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2295.327676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2295.400427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2304.634262] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 08:42:06 (1788871326) [ 2304.800267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2304.822957] LNet: 113167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.828803] LNetError: 113167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2304.845478] LNet: Removed LNI 192.168.204.48@tcp [ 2304.859666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.902263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2304.938360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2305.716446] Key type .llcrypt unregistered [ 2305.726263] Key type ._llcrypt unregistered [ 2318.407775] Key type ._llcrypt registered [ 2318.410292] Key type .llcrypt registered [ 2320.073580] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 08:42:22 (1788871342) [ 2320.466750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2320.494760] alg: No test for adler32 (adler32-zlib) [ 2321.473954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2321.588363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2321.617647] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2322.566279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2322.638433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2330.847681] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 08:42:33 (1788871353) [ 2330.953181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2330.980839] LNet: 114717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2330.990983] LNetError: 114717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2332.006850] LNet: Removed LNI 192.168.204.48@tcp [ 2332.053953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2332.128226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2332.175309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2337.916148] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 08:42:40 (1788871360) [ 2338.211377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2338.231149] LNet: 115304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2338.272671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2338.350979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2338.770149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2338.817558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2338.887824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2339.445762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2339.496882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2339.620341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.658673] LNet: 115337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.681545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2339.750847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2340.284648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2340.317917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2340.371706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2340.877970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2340.904129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2347.573343] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 08:42:50 (1788871370) [ 2347.773486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2347.841282] LNet: 115937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2347.870197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.951810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2347.979907] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2349.055076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2349.109814] LNet: Added LNI 192.168.204.48@tcp2 [8/256/0/180] [ 2350.206294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2350.343446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2350.631355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2350.653462] LNetError: 116217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2350.661413] LNet: Removed LNI 192.168.204.48@tcp [ 2350.670037] LNet: Removed LNI 192.168.204.48@tcp2 [ 2350.722736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.820531] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2350.831143] LNet: Added LNI 192.168.204.48@tcp2 [8/256/0/180] [ 2350.838236] LNet: 116223:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2351.471330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2351.727515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2351.750114] LNetError: 116260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2351.761599] LNetError: 116260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2351.767158] LNet: Removed LNI 192.168.204.48@tcp [ 2351.789266] LNet: Removed LNI 192.168.204.48@tcp2 [ 2351.841937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2351.951565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2351.988612] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2352.913619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2352.943477] LNet: Added LNI 192.168.204.48@tcp2 [8/256/0/180] [ 2353.876505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2353.936381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2354.001326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2354.351708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.379739] LNet: 116544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.384444] LNet: 116544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2354.392823] LNetError: 116544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2354.398302] LNetError: 116544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2355.424262] LNet: Removed LNI 192.168.204.48@tcp [ 2356.478981] LNet: Removed LNI 192.168.204.48@tcp2 [ 2356.579083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2356.678731] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2356.698459] LNet: Added LNI 192.168.204.48@tcp2 [8/256/0/180] [ 2356.711074] LNet: 116550:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2357.191325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2357.467486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2357.488452] LNet: Removed LNI 192.168.204.48@tcp [ 2358.514386] LNet: Removed LNI 192.168.204.48@tcp2 [ 2358.558499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.630733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2358.647798] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2359.464804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2359.490724] LNet: Added LNI 192.168.204.48@tcp2 [8/256/0/180] [ 2360.322903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2360.408542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2360.737932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2360.783874] LNetError: 116867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2360.794898] LNetError: 116867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2360.804500] LNet: Removed LNI 192.168.204.48@tcp [ 2360.818600] LNet: Removed LNI 192.168.204.48@tcp2 [ 2360.847852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.944222] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2360.958379] LNet: Added LNI 192.168.204.48@tcp2 [8/256/0/180] [ 2360.963191] LNet: 116873:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2361.741640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2361.904466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2362.976261] LNet: Removed LNI 192.168.204.48@tcp [ 2362.986618] LNet: Removed LNI 192.168.204.48@tcp2 [ 2363.033486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2363.123997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2363.179559] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2363.990447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2364.017435] LNet: Added LNI 192.168.204.48@tcp2 [8/256/0/180] [ 2364.944380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2364.979633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2365.031605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2365.249450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.267907] LNet: 117195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2365.275732] LNet: 117195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2366.305251] LNet: Removed LNI 192.168.204.48@tcp [ 2367.336727] LNet: Removed LNI 192.168.204.48@tcp2 [ 2367.396837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2367.498693] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2367.513414] LNet: Added LNI 192.168.204.48@tcp2 [8/256/0/180] [ 2367.524411] LNet: 117201:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2368.022291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2368.228422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2368.249549] LNet: Removed LNI 192.168.204.48@tcp [ 2369.320800] LNet: Removed LNI 192.168.204.48@tcp2 [ 2369.368905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.459526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2369.494752] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2370.154185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2370.182401] LNet: Added LNI 192.168.204.48@tcp2 [8/256/0/180] [ 2370.862524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2370.890661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2370.938730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2371.235606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2371.255662] LNetError: 117534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2371.263201] LNetError: 117534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2371.269261] LNet: Removed LNI 192.168.204.48@tcp [ 2371.275853] LNet: Removed LNI 192.168.204.48@tcp2 [ 2371.301136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.353245] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2371.364692] LNet: Added LNI 192.168.204.48@tcp2 [8/256/0/180] [ 2371.749047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2372.094279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2373.152502] LNet: Removed LNI 192.168.204.48@tcp [ 2373.164301] LNet: Removed LNI 192.168.204.48@tcp2 [ 2373.192733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2373.256503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2373.284336] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2374.021731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2374.038138] LNet: Added LNI 192.168.204.48@tcp2 [8/256/0/180] [ 2374.888063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2374.950329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2374.999732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2375.076642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2375.376771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2376.416246] LNet: Removed LNI 192.168.204.48@tcp [ 2377.453053] LNet: Removed LNI 192.168.204.48@tcp2 [ 2377.511854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2377.595608] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2377.604603] LNet: Added LNI 192.168.204.48@tcp2 [8/256/0/180] [ 2377.612634] LNet: 117900:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2377.625255] LNet: 117900:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 2378.196502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2385.667865] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2387.210768] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 08:43:29 (1788871409) [ 2387.490336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2387.536396] LNet: 118659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2387.543267] LNet: 118659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2387.558594] LNetError: 118659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2387.564173] LNetError: 118659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2387.573125] LNet: Removed LNI 192.168.204.48@tcp [ 2388.590142] LNet: Removed LNI 192.168.204.48@tcp2 [ 2388.632658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2389.073075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2389.104318] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2389.872744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2389.895162] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2390.773671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2390.794502] LNet: Removed LNI 10.1.2.3@tcp [ 2390.803746] LNet: Removed LNI 192.168.204.48@tcp [ 2390.819568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2390.897240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2390.924171] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2390.931942] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2391.414315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2391.440756] LNet: Removed LNI 192.168.204.48@tcp [ 2391.455892] LNet: Removed LNI 10.1.2.3@tcp [ 2397.711914] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 08:43:40 (1788871420) [ 2398.128712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2398.167137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2398.297376] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2398.491468] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2398.504691] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2398.516587] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2400.917198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2400.948654] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2402.000110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2402.045801] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2403.757184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.48@tcp [ 2403.823679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2403.893311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.48@tcp -m PUT -e local_timeout -r 1 [ 2403.960587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2404.017976] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2404.026445] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2404.265443] LNetError: 119602:0:(lib-move.c:4766:lnet_parse()) 192.168.204.48@tcp, src 192.168.204.48@tcp: Dropping PUT (error -108 looking up sender) [ 2406.305823] LNet: Removed LNI 10.1.2.3@tcp [ 2407.400902] LNet: Removed LNI 192.168.204.48@tcp [ 2407.867339] Key type .llcrypt unregistered [ 2407.871710] Key type ._llcrypt unregistered [ 2420.387237] Key type ._llcrypt registered [ 2420.390392] Key type .llcrypt registered [ 2421.730568] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 08:44:04 (1788871444) [ 2422.116992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2422.135148] alg: No test for adler32 (adler32-zlib) [ 2423.026494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2423.365444] LNet: 120914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2423.930823] Key type .llcrypt unregistered [ 2423.933506] Key type ._llcrypt unregistered [ 2430.627538] Key type ._llcrypt registered [ 2430.637701] Key type .llcrypt registered [ 2430.914738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2430.932276] alg: No test for adler32 (adler32-zlib) [ 2431.862669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2431.898979] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2439.806948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.148@tcp [ 2440.014388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2440.521847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.148@tcp [ 2440.736819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2440.763566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2440.793293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2440.820979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2440.852947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2440.876757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2440.902199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2440.926437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2440.941785] LNet: 121364:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2440.955463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2440.983508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2441.011514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2441.038771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2441.062861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2441.087758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2441.759684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2441.974949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2442.008896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2442.055371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2442.092633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2442.140716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2442.150149] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.148@tcp [ 2442.191577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2442.234352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2442.266348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2442.288825] LNet: 121481:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2442.316259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2442.366346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2442.450121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2442.487710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2442.525685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2442.568699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2443.388604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.148@tcp [ 2443.675720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2443.710994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2443.773346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2443.848778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2443.911847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2443.955975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2444.005105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2444.060943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2444.100426] LNet: 121598:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2444.144329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2444.257631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2444.327349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2444.379919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2444.461147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2444.529933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2445.378689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2445.672626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2445.712353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2445.737628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2445.761171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2445.815122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2445.877871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2445.957249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2446.024949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2446.059851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2446.132740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2446.211232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2446.288984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2446.330883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2446.384670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2446.860737] LNet: 121762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2446.871655] LNetError: 121762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2446.900233] LNet: Removed LNI 192.168.204.48@tcp [ 2447.310330] Key type .llcrypt unregistered [ 2447.315732] Key type ._llcrypt unregistered [ 2466.995758] Key type ._llcrypt registered [ 2466.998044] Key type .llcrypt registered [ 2468.250508] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 08:44:50 (1788871490) [ 2468.695555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2468.722450] alg: No test for adler32 (adler32-zlib) [ 2469.714116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2470.183070] LNet: 122649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2471.043702] Key type .llcrypt unregistered [ 2471.048143] Key type ._llcrypt unregistered [ 2477.778425] Key type ._llcrypt registered [ 2477.789246] Key type .llcrypt registered [ 2478.125845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2478.142581] alg: No test for adler32 (adler32-zlib) [ 2479.220552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2479.299195] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2489.267480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.148@tcp [ 2499.168256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2499.218926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2499.244785] LNet: Added LNI 192.168.204.48@tcp1 [8/256/0/180] [ 2500.045158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2500.605659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.148@tcp [ 2500.833343] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.148@tcp [ 2500.890875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2500.954767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2500.993558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2501.030242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2501.092636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2501.155710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2501.220426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2501.250832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2501.286889] LNet: 123301:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2501.324637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2501.384322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2501.434177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2501.461473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2501.510564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2501.554350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2502.080718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2502.260791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2502.280545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2502.307559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2502.328726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2502.354102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2502.385495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2502.423239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2502.453399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2502.469276] LNet: 123419:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2502.485476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2502.511318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2502.539807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2502.576770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2502.608160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2502.651511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2503.182324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.148@tcp [ 2503.378692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2503.404216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2503.427441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2503.451212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2503.478652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2503.506463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2503.527941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2503.556608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2503.572526] LNet: 123536:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2503.594967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2503.626692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2503.661830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2503.689784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2503.719611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2503.745443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2504.284173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2504.435488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2504.480857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2504.522136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2504.572630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2504.621394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2504.677252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2504.739811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2504.779763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2504.814435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2504.851179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2504.885346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2504.955073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2505.024102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2505.082875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2509.524137] LNet: 123734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2509.534611] LNetError: 123734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2509.545964] LNet: Removed LNI 192.168.204.48@tcp [ 2509.572687] LNet: Removed LNI 192.168.204.48@tcp1 [ 2510.040394] Key type .llcrypt unregistered [ 2510.043237] Key type ._llcrypt unregistered [ 2527.089256] Key type ._llcrypt registered [ 2527.090957] Key type .llcrypt registered [ 2528.443459] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 08:45:50 (1788871550) [ 2529.911664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2531.932370] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 08:45:54 (1788871554) [ 2539.854186] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 08:46:02 (1788871562) [ 2547.546705] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 08:46:10 (1788871570) [ 2555.778964] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 08:46:18 (1788871578) [ 2563.987496] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 08:46:26 (1788871586) [ 2572.723499] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 08:46:34 (1788871594) [ 2580.429170] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 08:46:43 (1788871603) [ 2588.284634] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 08:46:50 (1788871610) [ 2595.390298] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 08:46:57 (1788871617) [ 2602.030916] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 08:47:04 (1788871624) [ 2608.404054] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 08:47:11 (1788871631) [ 2614.527684] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 08:47:17 (1788871637) [ 2620.692690] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 08:47:23 (1788871643) [ 2626.347449] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 08:47:29 (1788871649) [ 2632.668873] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 08:47:35 (1788871655) [ 2640.519596] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 08:47:43 (1788871663) [ 2647.054510] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 08:47:49 (1788871669) [ 2653.984368] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 08:47:56 (1788871676) [ 2654.457979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2654.491035] alg: No test for adler32 (adler32-zlib) [ 2655.414281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2655.544647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2655.565831] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2656.713588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2656.742226] LNet: 134737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2656.752679] LNetError: 134737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2656.789795] LNet: Removed LNI 192.168.204.48@tcp [ 2656.812615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2656.880689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2656.896307] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2663.775574] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 08:48:06 (1788871686) [ 2671.694566] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 08:48:13 (1788871693) [ 2679.347564] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 08:48:22 (1788871702) [ 2686.387513] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 08:48:29 (1788871709) [ 2692.889193] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 08:48:35 (1788871715) [ 2699.631592] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 08:48:42 (1788871722) [ 2706.091836] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 08:48:48 (1788871728) [ 2713.040289] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 08:48:55 (1788871735) [ 2721.106315] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 08:49:03 (1788871743) [ 2727.669463] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 08:49:10 (1788871750) [ 2734.067258] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 08:49:16 (1788871756) [ 2740.945072] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 08:49:23 (1788871763) [ 2748.066184] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2412 sec ========== 08:49:30 (1788871770) [ 2748.434986] LNet: 142200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2748.445502] LNetError: 142200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2748.470177] LNet: Removed LNI 192.168.204.48@tcp [ 2749.178566] Key type .llcrypt unregistered [ 2749.183674] Key type ._llcrypt unregistered [ 2760.861296] Key type ._llcrypt registered [ 2760.863013] Key type .llcrypt registered [ 2761.330579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2761.340383] alg: No test for adler32 (adler32-zlib) [ 2762.531165] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 2762.768602] LNet: Added LNI 192.168.204.48@tcp [8/256/0/180] [ 2764.512251] Key type lgssc registered [ 2765.617810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2826.126848] Lustre: Mounted lustre-client [ 2830.985685] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2838.642563] Lustre: Unmounted lustre-client [ 2854.548849] Key type lgssc unregistered [ 2854.810702] LNet: 145129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2854.820510] LNetError: 145129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2854.839857] LNet: Removed LNI 192.168.204.48@tcp [ 2855.478190] Key type .llcrypt unregistered [ 2855.480768] Key type ._llcrypt unregistered