[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 465739287 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003328] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008285] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011120] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.015037] *** VALIDATE selinux *** [ 0.023452] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027786] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029041] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031099] *** VALIDATE tmpfs *** [ 0.032396] *** VALIDATE proc *** [ 0.033211] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035248] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038035] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042046] debug: unmapping init [mem 0xffffffff9d659000-0xffffffff9d660fff] [ 0.044946] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045687] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.054241] rcu: Hierarchical SRCU implementation. [ 0.056418] smp: Bringing up secondary CPUs ... [ 0.057495] x86: Booting SMP configuration: [ 0.058022] .... node #0, CPUs: #1 #2 #3 [ 0.061597] smp: Brought up 1 node, 4 CPUs [ 0.063019] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.269585] node 0 deferred pages initialised in 203ms [ 0.271151] devtmpfs: initialized [ 0.272337] x86/mm: Memory block size: 128MB [ 0.276237] gcov: version magic: 0x41383552 [ 0.279417] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.283156] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.286453] pinctrl core: initialized pinctrl subsystem [ 0.288227] [ 0.288778] ************************************************************* [ 0.291018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.294019] ** ** [ 0.296016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.298015] ** ** [ 0.301019] ** This means that this kernel is built to expose internal ** [ 0.303015] ** IOMMU data structures, which may compromise security on ** [ 0.305019] ** your system. ** [ 0.308018] ** ** [ 0.310017] ** If you see this message and you are not debugging the ** [ 0.313019] ** kernel, report this immediately to your vendor! ** [ 0.315016] ** ** [ 0.317022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.320018] ************************************************************* [ 0.322800] NET: Registered protocol family 16 [ 0.324536] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.327082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.330080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.334054] cpuidle: using governor menu [ 0.335840] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.336745] PCI: Using configuration type 1 for base access [ 0.338136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.348157] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.351053] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.354080] cryptd: max_cpu_qlen set to 1000 [ 0.356287] ACPI: Added _OSI(Module Device) [ 0.358027] ACPI: Added _OSI(Processor Device) [ 0.360015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.361013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.366285] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.372359] ACPI: Interpreter enabled [ 0.373052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.374016] ACPI: Using IOAPIC for interrupt routing [ 0.376101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.379414] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.389648] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.392044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.394019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.397084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.402318] acpiphp: Slot [2] registered [ 0.404180] acpiphp: Slot [5] registered [ 0.405138] acpiphp: Slot [6] registered [ 0.406092] acpiphp: Slot [3] registered [ 0.407141] acpiphp: Slot [4] registered [ 0.409169] acpiphp: Slot [7] registered [ 0.411184] acpiphp: Slot [8] registered [ 0.412396] acpiphp: Slot [9] registered [ 0.414158] acpiphp: Slot [10] registered [ 0.416125] acpiphp: Slot [11] registered [ 0.418109] acpiphp: Slot [12] registered [ 0.419088] acpiphp: Slot [13] registered [ 0.420088] acpiphp: Slot [14] registered [ 0.422128] acpiphp: Slot [15] registered [ 0.423180] acpiphp: Slot [16] registered [ 0.425171] acpiphp: Slot [17] registered [ 0.427184] acpiphp: Slot [18] registered [ 0.429167] acpiphp: Slot [19] registered [ 0.430139] acpiphp: Slot [20] registered [ 0.432147] acpiphp: Slot [21] registered [ 0.433134] acpiphp: Slot [22] registered [ 0.434150] acpiphp: Slot [23] registered [ 0.436148] acpiphp: Slot [24] registered [ 0.438182] acpiphp: Slot [25] registered [ 0.439173] acpiphp: Slot [26] registered [ 0.441118] acpiphp: Slot [27] registered [ 0.443140] acpiphp: Slot [28] registered [ 0.444176] acpiphp: Slot [29] registered [ 0.446170] acpiphp: Slot [30] registered [ 0.448156] acpiphp: Slot [31] registered [ 0.449084] PCI host bridge to bus 0000:00 [ 0.451032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.453049] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.455023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.457019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.459026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.461024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.462152] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.465701] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.468375] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.475732] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.480056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.483026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.484022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.486023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.489408] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.490000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.492050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.494798] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.499014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.509015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.511017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.517286] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.525022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.529985] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.539877] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.547056] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.554021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.561021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.576029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.586166] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.589399] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.591394] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.593370] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.596221] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.600154] iommu: Default domain type: Passthrough [ 0.602380] SCSI subsystem initialized [ 0.603118] ACPI: bus type USB registered [ 0.604136] usbcore: registered new interface driver usbfs [ 0.606076] usbcore: registered new interface driver hub [ 0.607093] usbcore: registered new device driver usb [ 0.609182] pps_core: LinuxPPS API ver. 1 registered [ 0.611012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.613066] PTP clock support registered [ 0.614203] EDAC MC: Ver: 3.0.0 [ 0.616171] PCI: Using ACPI for IRQ routing [ 0.618983] NetLabel: Initializing [ 0.620024] NetLabel: domain hash size = 128 [ 0.621014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.625118] NetLabel: unlabeled traffic allowed by default [ 0.627189] vgaarb: loaded [ 0.629168] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.630016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.634483] clocksource: Switched to clocksource kvm-clock [ 0.749610] VFS: Disk quotas dquot_6.6.0 [ 0.751605] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.754710] *** VALIDATE ramfs *** [ 0.756170] *** VALIDATE hugetlbfs *** [ 0.758813] pnp: PnP ACPI init [ 0.761652] pnp: PnP ACPI: found 6 devices [ 0.780327] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.784195] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.786306] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.788695] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.791219] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.793761] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.796675] NET: Registered protocol family 2 [ 0.799441] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.804535] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.808163] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.813650] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.817282] TCP: Hash tables configured (established 65536 bind 65536) [ 0.820657] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.824092] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.827401] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.830726] NET: Registered protocol family 1 [ 0.836571] RPC: Registered named UNIX socket transport module. [ 0.838592] RPC: Registered udp transport module. [ 0.840365] RPC: Registered tcp transport module. [ 0.842131] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.844313] NET: Registered protocol family 44 [ 0.846349] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.848688] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.850832] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.852994] PCI: CLS 0 bytes, default 64 [ 0.854674] Unpacking initramfs... [ 2.277626] debug: unmapping init [mem 0xffff981b3cc64000-0xffff981b3ffcffff] [ 2.281620] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.283647] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.286406] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.802309] Initialise system trusted keyrings [ 2.803997] Key type blacklist registered [ 2.806512] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.816122] zbud: loaded [ 2.819425] *** VALIDATE nfs *** [ 2.820776] *** VALIDATE nfs4 *** [ 2.822580] pstore: using deflate compression [ 2.827148] Platform Keyring initialized [ 2.929741] NET: Registered protocol family 38 [ 2.931682] Key type asymmetric registered [ 2.933167] Asymmetric key parser 'x509' registered [ 2.935030] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.938109] io scheduler mq-deadline registered [ 2.939571] io scheduler kyber registered [ 2.941170] io scheduler bfq registered [ 2.942970] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.945923] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.948909] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.951384] ACPI: Power Button [PWRF] [ 2.956152] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.962350] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.972866] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.002591] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.038163] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.043785] Non-volatile memory driver v1.3 [ 3.045527] Linux agpgart interface v0.103 [ 3.079549] virtio_blk virtio1: [vda] 149784 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.082927] vda: detected capacity change from 0 to 76689408 [ 3.096939] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.099684] vdb: detected capacity change from 0 to 1073741824 [ 3.107108] libphy: Fixed MDIO Bus: probed [ 3.116255] usbcore: registered new interface driver usbserial_generic [ 3.119230] usbserial: USB Serial support registered for generic [ 3.122365] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.128128] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.129837] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.132563] mousedev: PS/2 mouse device common for all mice [ 3.135460] rtc_cmos 00:05: RTC can wake from S4 [ 3.139328] rtc_cmos 00:05: registered as rtc0 [ 3.139597] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.141779] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.147554] intel_pstate: CPU model not supported [ 3.150397] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.152284] hid: raw HID events driver (C) Jiri Kosina [ 3.156126] usbcore: registered new interface driver usbhid [ 3.156287] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.158047] usbhid: USB HID core driver [ 3.162701] drop_monitor: Initializing network drop monitor service [ 3.165122] Initializing XFRM netlink socket [ 3.167312] NET: Registered protocol family 10 [ 3.170622] Segment Routing with IPv6 [ 3.172088] NET: Registered protocol family 17 [ 3.174273] mpls_gso: MPLS GSO support [ 3.179874] RAS: Correctable Errors collector initialized. [ 3.181727] AVX version of gcm_enc/dec engaged. [ 3.183176] AES CTR mode by8 optimization enabled [ 3.259686] sched_clock: Marking stable (3259643898, 0)->(4159543079, -899899181) [ 3.263057] registered taskstats version 1 [ 3.265051] Loading compiled-in X.509 certificates [ 3.266964] zswap: loaded using pool lzo/zbud [ 3.293502] Key type big_key registered [ 3.305821] Key type encrypted registered [ 3.307323] ima: No TPM chip found, activating TPM-bypass! [ 3.309223] ima: Allocated hash algorithm: sha1 [ 3.310600] ima: No architecture policies found [ 3.312093] evm: Initialising EVM extended attributes: [ 3.313829] evm: security.selinux [ 3.315012] evm: security.ima [ 3.315995] evm: security.capability [ 3.317213] evm: HMAC attrs: 0x1 [ 3.319256] rtc_cmos 00:05: setting system clock to 2026-09-06 03:06:49 UTC (1788664009) [ 3.325391] debug: unmapping init [mem 0xffffffff9e603000-0xffffffff9e7fffff] [ 3.328615] debug: unmapping init [mem 0xffffffff9d382000-0xffffffff9d658fff] [ 3.338095] Write protecting the kernel read-only data: 28672k [ 3.341155] debug: unmapping init [mem 0xffffffff9ba03000-0xffffffff9bbfffff] [ 3.343917] debug: unmapping init [mem 0xffffffff9c314000-0xffffffff9c3fffff] [ 3.378800] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.386799] systemd[1]: Detected virtualization kvm. [ 3.388939] systemd[1]: Detected architecture x86-64. [ 3.390986] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.423338] systemd[1]: No hostname configured. [ 3.425095] systemd[1]: Set hostname to . [ 3.427160] random: systemd: uninitialized urandom read (16 bytes read) [ 3.429507] systemd[1]: Initializing machine ID from random generator. [ 3.486863] random: ln: uninitialized urandom read (6 bytes read) [ 3.575980] random: systemd: uninitialized urandom read (16 bytes read) [ 3.578655] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.583198] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.587405] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.275826] device-mapper: uevent: version 1.0.3 [ 4.277881] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.994244] virtio_net virtio0 ens2: renamed from eth0 [ 5.057186] scsi host0: ata_piix [ 5.086227] scsi host1: ata_piix [ 5.087833] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.090367] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.504498] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.822682] random: crng init done [ 9.824055] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.282084] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.511186] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.773068] SELinux: Disabled at runtime. [ 11.830993] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.839786] systemd[1]: Detected virtualization kvm. [ 11.841891] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.396518] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.400365] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.406482] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.412302] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.417754] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.425645] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.440922] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ 12.500505] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.917369] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.237254] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.308768] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.465939] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.478063] EDAC sbridge: Ver: 1.1.2 [ 14.842698] Key type dns_resolver registered [ 15.154988] NFS: Registering the id_resolver key type [ 15.157037] Key type id_resolver registered [ 15.158651] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg421-client login: [ 57.006424] libcfs: loading out-of-tree module taints kernel. [ 57.089204] Key type ._llcrypt registered [ 57.094663] Key type .llcrypt registered [ 58.035799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 58.054518] alg: No test for adler32 (adler32-zlib) [ 59.610450] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 60.525593] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 62.424639] Key type lgssc registered [ 64.835844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 115.151214] hrtimer: interrupt took 6154657 ns [ 208.519364] Lustre: Mounted lustre-client - version 2.17.58_4_g3b30273 [ 214.447669] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 230.279529] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing check_logdir /tmp/testlogs/ [ 233.955069] Lustre: lustre-OST0000-osc-ffff981b87027000: disconnect after 23s idle [ 235.555481] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing yml_node [ 241.236364] Lustre: DEBUG MARKER: Client: 2.17.58.4 [ 244.505661] Lustre: DEBUG MARKER: MDS: 2.17.58.4 [ 247.134954] Lustre: DEBUG MARKER: OSS: 2.17.58.4 [ 249.080622] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Sep 5 23:10:53 EDT 2026 [ 266.689930] Lustre: Unmounted lustre-client [ 299.965713] Key type lgssc unregistered [ 300.394158] LNet: 8408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.409070] LNetError: 8408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 300.430211] LNet: Removed LNI 192.168.204.21@tcp [ 301.599355] Key type .llcrypt unregistered [ 301.601534] Key type ._llcrypt unregistered [ 335.976600] Key type ._llcrypt registered [ 335.979647] Key type .llcrypt registered [ 336.183921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 336.206548] alg: No test for adler32 (adler32-zlib) [ 337.133388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 337.225976] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 337.285902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 342.303836] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 342.473116] LNet: 9286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.489526] LNetError: 9286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 342.529168] LNet: Removed LNI 192.168.204.21@tcp [ 343.282810] Key type .llcrypt unregistered [ 343.285531] Key type ._llcrypt unregistered [ 351.262450] Key type ._llcrypt registered [ 351.272429] Key type .llcrypt registered [ 352.792521] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 23:12:37 (1788664357) [ 353.327553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 353.363644] alg: No test for adler32 (adler32-zlib) [ 354.393741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.474196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 354.496810] LNet: 9700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 354.516565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.589642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 354.618316] LNet: 9707:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 362.276384] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 23:12:46 (1788664366) [ 362.398282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.422515] LNet: 10284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.481587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.563668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 370.288489] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 23:12:55 (1788664375) [ 370.675911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.706935] LNet: 10872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.750757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.855660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 378.792442] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 23:13:03 (1788664383) [ 378.970431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.001669] LNet: 11454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.048824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.148983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 386.337321] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 23:13:11 (1788664391) [ 386.563686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.589860] LNet: 12043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.614877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.746560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 387.308291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 395.076924] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 23:13:20 (1788664400) [ 395.469584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.501405] LNet: 12633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.530188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.631655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 396.324046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 396.990301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 405.334253] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 23:13:30 (1788664410) [ 405.607537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.663812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.740849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 412.569635] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 23:13:37 (1788664417) [ 412.668394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.684311] LNet: 13808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.694683] LNet: 13808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 412.743450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.835586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 412.902390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 413.625975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 413.691579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 414.447948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 414.525708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 415.350153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 415.375766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 415.615218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 415.658749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 416.215216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 416.284611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 417.126622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 417.177887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 418.202967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 418.274887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 419.114732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 419.201948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 420.113404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 420.149708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 421.099947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 421.154173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 422.281855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 422.396148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 423.170992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 423.213944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 434.345613] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 23:13:58 (1788664438) [ 434.507917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.611097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.817808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 434.856409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 442.919392] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 23:14:06 (1788664446) [ 443.065286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.164187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.320753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 443.417022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 451.810098] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 23:14:16 (1788664456) [ 452.092937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.118838] LNet: 15671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.124295] LNet: 15671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 452.142505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.213287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 452.244873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 459.992570] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 23:14:24 (1788664464) [ 460.159536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.183521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.244525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 460.281584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 468.985563] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 23:14:33 (1788664473) [ 469.136497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.216277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.421488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 469.503171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 478.009264] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 23:14:42 (1788664482) [ 478.112714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.174295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.390526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 478.457625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 487.286416] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 23:14:51 (1788664491) [ 487.442390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.494028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.568750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 488.145182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 495.808928] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 23:15:00 (1788664500) [ 496.010420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.092142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.311761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 496.536833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 504.973309] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 23:15:09 (1788664509) [ 505.078203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.127989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.305642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 505.576494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 512.417632] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 23:15:17 (1788664517) [ 512.600703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.687570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.190092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 513.462188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 520.351488] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 23:15:25 (1788664525) [ 520.504670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.550838] LNet: 20503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 520.567988] LNet: 20503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 520.599539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.664460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 520.710795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 527.544163] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 23:15:32 (1788664532) [ 527.802811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.870853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.945577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 528.034622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 535.055619] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 23:15:39 (1788664539) [ 535.145154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.209261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.312115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 542.802043] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 23:15:47 (1788664547) [ 543.243310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.281526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.355981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 550.060905] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 23:15:54 (1788664554) [ 550.343359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.400709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.466128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 550.935736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 557.964549] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 23:16:02 (1788664562) [ 558.272584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.331392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.428190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 558.481447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 566.121624] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 23:16:10 (1788664570) [ 566.492795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.563784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.674721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 566.724876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 573.761522] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 23:16:18 (1788664578) [ 574.182544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.216945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.289891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 574.321558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 580.763037] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 23:16:25 (1788664585) [ 581.154560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.231745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.333670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 581.372751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 589.087599] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 23:16:33 (1788664593) [ 589.495493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.548542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.651473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 589.813658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 589.851727] LNetError: 25821:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 589.926431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 589.985028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 597.669625] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 23:16:42 (1788664602) [ 597.797970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.835528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.997627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.leWDYpXyiH [ 598.102738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.leWDYpXyiH [ 598.161912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.leWDYpXyiH [ 605.994332] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 23:16:50 (1788664610) [ 606.301495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.388241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.490779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 606.555313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 606.634457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 614.006254] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 23:16:58 (1788664618) [ 614.232677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.293859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.401601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 614.457168] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 616.003626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 617.153715] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 618.983586] Key type lgssc registered [ 620.415641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 625.169509] LNetError: 27583:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.21@tcp99 has no available nets [ 628.695658] Key type lgssc unregistered [ 629.006185] LNetError: 28690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.054687] LNet: Removed LNI 192.168.204.21@tcp [ 631.035973] Key type .llcrypt unregistered [ 631.040398] Key type ._llcrypt unregistered [ 639.131570] Key type ._llcrypt registered [ 639.137486] Key type .llcrypt registered [ 640.809651] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 23:17:25 (1788664645) [ 641.766290] Key type .llcrypt unregistered [ 641.768130] Key type ._llcrypt unregistered [ 649.263752] Key type ._llcrypt registered [ 649.267063] Key type .llcrypt registered [ 649.484755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 649.501977] alg: No test for adler32 (adler32-zlib) [ 650.422627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.516592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 650.548952] LNet: 29601:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 650.559776] LNet: 29601:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 650.582395] LNet: 29601:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 650.607037] LNet: Message forwarding will be enabled in 2s [ 653.727135] LNet: Message forwarding enabled [ 654.264287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 654.307378] LNet: Message forwarding will stop in 2s to 6s [ 657.830211] LNet: Message forwarding disabled [ 658.920186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 658.955957] LNet: 29700:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 658.966132] LNet: 29700:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 658.976408] LNet: 29700:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 659.000924] LNet: Message forwarding will be enabled in 2s [ 659.245966] LNet: 29713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.965380] Key type .llcrypt unregistered [ 659.972041] Key type ._llcrypt unregistered [ 673.656706] Key type ._llcrypt registered [ 673.662206] Key type .llcrypt registered [ 675.330988] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 23:17:59 (1788664679) [ 675.746581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 675.760775] alg: No test for adler32 (adler32-zlib) [ 676.693372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.803971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 684.423516] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 23:18:08 (1788664688) [ 684.527846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.569390] LNet: 31109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.624608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.705832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 684.748819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 684.785318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 684.823713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 684.857322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 684.888341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 684.923944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 684.969897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 685.004865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 685.064618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 685.131755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 685.172764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 685.249677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 685.323912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 685.414029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 685.481423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 685.537667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 685.575669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 685.613988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 685.643309] LNetError: 31160:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 692.754608] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 23:18:17 (1788664697) [ 693.138218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.164812] LNet: 31731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.187968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.295843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 700.760574] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 23:18:25 (1788664705) [ 702.850512] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg421-server [ 704.754857] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 23:18:29 (1788664709) [ 705.541711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.592090] LNet: 32662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.628498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.720885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 705.749385] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 706.812333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[22-26]@tcp [ 706.847710] LNet: 32795:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 706.917093] LNetError: 32666:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.22@tcp is being used as a gateway but routing feature is not turned on [ 706.928433] LNetError: 32666:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.22@tcp has gone from up to down [ 715.288181] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 23:18:39 (1788664719) [ 717.127856] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg421-server [ 719.068913] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 23:18:43 (1788664723) [ 719.366654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.399075] LNet: 33732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.412266] LNetError: 33732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 719.441043] LNet: Removed LNI 192.168.204.21@tcp [ 719.498816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.614886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 719.669906] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 720.390217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[22-27/2]@tcp [ 720.419917] LNet: 33849:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 720.428618] LNet: 33849:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 720.459962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[22-27/2]@tcp [ 720.461644] LNetError: 33736:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.24@tcp is being used as a gateway but routing feature is not turned on [ 720.489251] LNetError: 33736:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 720.500958] LNetError: 33736:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.24@tcp has gone from up to down [ 720.514749] LNetError: 33736:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 727.714596] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 23:18:52 (1788664732) [ 728.107419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.141687] LNet: 34442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.147846] LNetError: 34442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.175379] LNet: Removed LNI 192.168.204.21@tcp [ 728.233133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.358621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 728.427364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.495727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.601292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 728.678606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.723763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.772428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 728.860522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.902712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.990966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 729.090272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.140681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.210757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 729.316759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.358053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.466102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 729.555928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.591257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.663228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 729.753992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.795643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.867760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 729.954410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.982677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.081496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 730.196698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.238802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.322689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 730.415886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.466564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.541457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 730.604856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.642854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.721974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 737.989235] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 23:19:02 (1788664742) [ 740.304687] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg421-server [ 741.982824] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 23:19:06 (1788664746) [ 743.865419] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg421-server [ 745.633177] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 23:19:10 (1788664750) [ 745.886239] LNet: 35843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.894230] LNet: 35843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 746.574318] Key type .llcrypt unregistered [ 746.578290] Key type ._llcrypt unregistered [ 754.364188] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 754.586503] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 754.600235] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 754.613230] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 757.181726] Key type ._llcrypt registered [ 757.183711] Key type .llcrypt registered [ 757.344352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 757.359448] alg: No test for adler32 (adler32-zlib) [ 758.350469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.395598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 758.417333] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 759.239973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 759.268444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.849549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 761.888404] LNet: Removed LNI 10.1.2.3@tcp [ 762.041077] LNet: 36555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.052477] LNetError: 36555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.073209] LNet: Removed LNI 192.168.204.21@tcp [ 763.088633] Key type .llcrypt unregistered [ 763.091839] Key type ._llcrypt unregistered [ 775.722989] Key type ._llcrypt registered [ 775.724938] Key type .llcrypt registered [ 777.005519] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 23:19:41 (1788664781) [ 777.488705] Key type .llcrypt unregistered [ 777.494628] Key type ._llcrypt unregistered [ 783.321638] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 783.534650] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 783.548616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 783.559971] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 786.288893] Key type ._llcrypt registered [ 786.295935] Key type .llcrypt registered [ 786.504251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 786.527281] alg: No test for adler32 (adler32-zlib) [ 787.456313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.528812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 787.562603] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 788.367920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 788.392303] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 790.869620] LNet: 38044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.877561] LNetError: 38044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 790.881816] LNet: Removed LNI 10.1.2.3@tcp [ 790.890829] LNet: Removed LNI 192.168.204.21@tcp [ 791.609542] Key type .llcrypt unregistered [ 791.614079] Key type ._llcrypt unregistered [ 802.364653] Key type ._llcrypt registered [ 802.369319] Key type .llcrypt registered [ 803.644283] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 23:20:08 (1788664808) [ 804.196406] Key type .llcrypt unregistered [ 804.199182] Key type ._llcrypt unregistered [ 810.041521] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 810.232942] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 810.241539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 810.246573] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 812.862650] Key type ._llcrypt registered [ 812.866244] Key type .llcrypt registered [ 813.078666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 813.098667] alg: No test for adler32 (adler32-zlib) [ 814.071146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.115588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 814.142276] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 815.931634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 815.955065] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 817.716368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 817.733405] LNet: Removed LNI 10.1.2.3@tcp [ 817.745551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 817.775311] LNet: Removed LNI 10.1.2.31@tcp [ 817.793709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 817.826440] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 818.683709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 818.697688] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 819.534319] LNet: 39957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.542848] LNetError: 39957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 819.550345] LNet: Removed LNI 10.1.2.31@tcp [ 819.561675] LNet: Removed LNI 10.1.2.3@tcp [ 820.230868] Key type .llcrypt unregistered [ 820.232263] Key type ._llcrypt unregistered [ 831.353553] Key type ._llcrypt registered [ 831.356414] Key type .llcrypt registered [ 832.637025] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 23:20:37 (1788664837) [ 833.236646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 833.249648] alg: No test for adler32 (adler32-zlib) [ 834.212764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.290397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.323938] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 835.222129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.256525] LNet: 40906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.263829] LNetError: 40906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.285180] LNet: Removed LNI 192.168.204.21@tcp [ 835.308721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.371519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.21@tcp [ 835.393279] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 836.317867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.21@tcp [ 836.359160] LNet: Removed LNI 192.168.204.21@tcp [ 836.481348] LNet: 41041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.122804] Key type .llcrypt unregistered [ 837.125586] Key type ._llcrypt unregistered [ 848.354617] Key type ._llcrypt registered [ 848.357397] Key type .llcrypt registered [ 849.502477] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 23:20:54 (1788664854) [ 851.134645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg421-server [ 852.782334] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 23:20:57 (1788664857) [ 853.344265] Key type .llcrypt unregistered [ 853.348633] Key type ._llcrypt unregistered [ 859.464116] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 859.611540] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 859.613992] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 859.616621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 862.130495] Key type ._llcrypt registered [ 862.132830] Key type .llcrypt registered [ 862.382369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 862.414435] alg: No test for adler32 (adler32-zlib) [ 863.358730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.449965] LNet: Added LNI 192.168.204.21@tcp1 [8/256/0/180] [ 864.366161] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 866.730276] LNet: 42866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.737213] LNetError: 42866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.744475] LNet: Removed LNI 192.168.204.21@tcp1 [ 866.757244] LNet: Removed LNI 10.1.2.3@tcp2 [ 867.571641] Key type .llcrypt unregistered [ 867.577225] Key type ._llcrypt unregistered [ 881.006582] Key type ._llcrypt registered [ 881.009758] Key type .llcrypt registered [ 882.515244] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 23:21:26 (1788664886) [ 882.915474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 882.930852] alg: No test for adler32 (adler32-zlib) [ 883.895199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.971246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 883.994807] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 885.086059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.118993] LNet: 43832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.129853] LNetError: 43832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.151635] LNet: Removed LNI 192.168.204.21@tcp [ 885.186506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.233178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 885.282762] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 892.741815] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 23:21:37 (1788664897) [ 893.050100] LNet: 44423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.066412] LNetError: 44423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.111471] LNet: Removed LNI 192.168.204.21@tcp [ 893.954568] Key type .llcrypt unregistered [ 893.956767] Key type ._llcrypt unregistered [ 899.826536] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 899.967755] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 899.974586] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 899.984621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 902.747234] Key type ._llcrypt registered [ 902.751723] Key type .llcrypt registered [ 903.043752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 903.069705] alg: No test for adler32 (adler32-zlib) [ 903.986608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.042660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 904.074894] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 904.874661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 904.898595] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 907.349909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.369734] LNet: 45166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.375736] LNetError: 45166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 907.381095] LNet: Removed LNI 10.1.2.3@tcp [ 907.391603] LNet: Removed LNI 192.168.204.21@tcp [ 907.409817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.496831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 907.523933] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 907.529581] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 908.243559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.302860] LNet: 45193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.315764] LNetError: 45193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 908.330098] LNetError: 45193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 908.346876] LNet: Removed LNI 10.1.2.3@tcp [ 908.359719] LNet: Removed LNI 192.168.204.21@tcp [ 908.396173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.472391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 908.536577] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 908.553173] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 909.188902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 909.264646] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 910.465440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 910.483421] LNet: 45355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.488566] LNetError: 45355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 910.493085] LNetError: 45355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 910.497980] LNet: Removed LNI 10.1.2.3@tcp [ 910.500379] LNet: Removed LNI 192.168.204.21@tcp [ 910.512934] LNet: Removed LNI 192.168.204.21@tcp2 [ 910.538486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.616815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 910.652221] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 910.658323] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 910.662978] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 918.750058] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 23:22:03 (1788664923) [ 918.929278] LNet: 45951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.934533] LNetError: 45951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 918.942247] LNetError: 45951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 918.947673] LNet: Removed LNI 10.1.2.3@tcp [ 918.949100] LNet: Removed LNI 192.168.204.21@tcp [ 918.974340] LNet: Removed LNI 192.168.204.21@tcp2 [ 919.817145] Key type .llcrypt unregistered [ 919.824221] Key type ._llcrypt unregistered [ 926.181338] Key type ._llcrypt registered [ 926.183827] Key type .llcrypt registered [ 926.418192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 926.430995] alg: No test for adler32 (adler32-zlib) [ 927.371663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.442315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 927.467163] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 928.214621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 928.236533] LNet: 46345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.243912] LNetError: 46345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.257895] LNet: Removed LNI 192.168.204.21@tcp [ 928.286787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.352703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 928.370897] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 928.666164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 928.682146] LNet: Removed LNI 192.168.204.21@tcp [ 928.696226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.777442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 928.840477] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 929.311962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.327079] LNet: 46405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.331826] LNet: 46405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 929.336731] LNetError: 46405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.344337] LNetError: 46405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 929.356809] LNet: Removed LNI 192.168.204.21@tcp [ 929.375303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.424299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 929.442666] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 930.049178] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 930.216717] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 930.224772] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 930.242177] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 932.585460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.603525] LNet: 46483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.610609] LNetError: 46483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.628244] LNet: Removed LNI 192.168.204.21@tcp [ 932.649766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.720638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 932.749952] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 939.418242] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 23:22:24 (1788664944) [ 939.541870] LNet: 47076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.546652] LNetError: 47076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.573273] LNet: Removed LNI 192.168.204.21@tcp [ 940.132404] Key type .llcrypt unregistered [ 940.134957] Key type ._llcrypt unregistered [ 945.869883] Key type ._llcrypt registered [ 945.872395] Key type .llcrypt registered [ 946.109316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 946.128924] alg: No test for adler32 (adler32-zlib) [ 947.092044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.140394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 947.178536] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 947.916762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.952504] LNet: 47474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.965472] LNetError: 47474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.980027] LNet: Removed LNI 192.168.204.21@tcp [ 948.001980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.060575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 948.090760] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 954.701261] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 23:22:39 (1788664959) [ 954.922869] LNet: 48064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.930252] LNetError: 48064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.965512] LNet: Removed LNI 192.168.204.21@tcp [ 955.704970] Key type .llcrypt unregistered [ 955.709837] Key type ._llcrypt unregistered [ 962.464199] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 962.771976] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 962.786582] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 962.806082] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 965.455491] Key type ._llcrypt registered [ 965.458605] Key type .llcrypt registered [ 965.764104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 965.794430] alg: No test for adler32 (adler32-zlib) [ 966.723939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.836738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 966.865844] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 967.789027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.812498] LNet: 48514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.824617] LNetError: 48514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.842080] LNet: Removed LNI 192.168.204.21@tcp [ 967.874480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.962950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 968.014663] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 975.116809] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 23:22:59 (1788664979) [ 975.376553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.455294] LNet: 49112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.464081] LNetError: 49112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.500294] LNet: Removed LNI 192.168.204.21@tcp [ 975.525425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.730399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 981.819822] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 23:23:06 (1788664986) [ 981.947719] LNet: 49686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.570587] Key type .llcrypt unregistered [ 982.574852] Key type ._llcrypt unregistered [ 989.708141] Key type ._llcrypt registered [ 989.709603] Key type .llcrypt registered [ 990.030853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 990.069701] alg: No test for adler32 (adler32-zlib) [ 991.273805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.409786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 991.463355] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 992.716136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.730156] LNet: 50088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.739980] LNetError: 50088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.760702] LNet: Removed LNI 192.168.204.21@tcp [ 992.786183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.891386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 992.922655] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1001.241176] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 23:23:25 (1788665005) [ 1001.573240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1001.610609] LNet: 50682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.617654] LNetError: 50682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1001.633687] LNet: Removed LNI 192.168.204.21@tcp [ 1001.655531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1001.719362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1008.004099] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 23:23:32 (1788665012) [ 1009.527940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1011.240597] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 23:23:36 (1788665016) [ 1013.003314] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1014.734806] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 23:23:39 (1788665019) [ 1016.429766] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1018.049323] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 23:23:42 (1788665022) [ 1019.899710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1021.877589] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 23:23:46 (1788665026) [ 1023.216638] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1024.992948] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 23:23:49 (1788665029) [ 1025.414366] LNet: 52733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.073540] Key type .llcrypt unregistered [ 1026.079576] Key type ._llcrypt unregistered [ 1032.503494] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1032.712143] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1032.715642] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1032.719216] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1035.258866] Key type ._llcrypt registered [ 1035.262209] Key type .llcrypt registered [ 1035.464390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1035.482631] alg: No test for adler32 (adler32-zlib) [ 1036.387927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.472455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1036.506199] LNet: Added LNI 192.168.204.21@tcp [8/512/0/180] [ 1036.515062] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1036.519153] LNet: 53054:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1036.645543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1036.672260] LNet: 53073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.681638] LNetError: 53073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1036.698246] LNet: Removed LNI 10.1.2.3@tcp [ 1036.702616] LNetError: 53073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.733541] LNet: Removed LNI 192.168.204.21@tcp [ 1036.753197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.847687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1036.879855] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1036.973198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1036.995958] LNet: 53097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.010960] LNetError: 53097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.044513] LNet: Removed LNI 192.168.204.21@tcp [ 1037.092456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.163220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1037.201072] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1037.210671] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1037.212615] LNet: 53105:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1037.324181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1037.360992] LNet: 53123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.369352] LNetError: 53123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1037.374491] LNet: Removed LNI 10.1.2.3@tcp [ 1037.378476] LNetError: 53123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.414121] LNet: Removed LNI 192.168.204.21@tcp [ 1037.449725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.547690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1037.594863] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1037.601805] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1037.603964] LNet: 53131:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1037.831779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1037.862539] LNet: 53157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.873782] LNetError: 53157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1037.879602] LNet: Removed LNI 10.1.2.3@tcp [ 1037.883839] LNetError: 53157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.915348] LNet: Removed LNI 192.168.204.21@tcp [ 1037.946922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1038.080726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1038.116274] LNet: Added LNI 192.168.204.21@tcp [8/256/0/182] [ 1038.120598] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1038.256249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1038.272778] LNet: 53185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.287423] LNetError: 53185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1038.302399] LNet: Removed LNI 10.1.2.3@tcp [ 1038.307603] LNetError: 53185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.327282] LNet: Removed LNI 192.168.204.21@tcp [ 1038.370896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1038.448697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1038.468798] LNet: Added LNI 192.168.204.21@tcp [8/256/0/182] [ 1038.474125] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1038.477806] LNet: 53193:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1038.574283] LNet: Removed LNI 10.1.2.3@tcp [ 1038.587701] LNet: Removed LNI 192.168.204.21@tcp [ 1039.267150] Key type .llcrypt unregistered [ 1039.268617] Key type ._llcrypt unregistered [ 1052.693716] Key type ._llcrypt registered [ 1052.699457] Key type .llcrypt registered [ 1053.932146] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 23:24:18 (1788665058) [ 1055.506492] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1057.521539] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 23:24:22 (1788665062) [ 1058.179275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1058.223365] alg: No test for adler32 (adler32-zlib) [ 1059.100911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1059.199765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1059.237341] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1060.005365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.21 [ 1060.020861] LNet: 54456:0:(config.c:1737:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1060.028817] LNet: Removed LNI 192.168.204.21@tcp [ 1068.203694] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 23:24:32 (1788665072) [ 1068.317799] LNet: 55019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.324547] LNetError: 55019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1068.346524] LNet: Removed LNI 192.168.204.21@tcp [ 1069.096113] Key type .llcrypt unregistered [ 1069.097818] Key type ._llcrypt unregistered [ 1076.787462] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1077.140205] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1080.111314] Key type ._llcrypt registered [ 1080.115770] Key type .llcrypt registered [ 1080.430194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1080.459911] alg: No test for adler32 (adler32-zlib) [ 1082.177648] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1091.223794] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 23:24:55 (1788665095) [ 1091.489964] LNet: 55934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.514027] LNetError: 55934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1091.551137] LNet: Removed LNI 10.1.2.3@tcp [ 1092.600680] Key type .llcrypt unregistered [ 1092.606784] Key type ._llcrypt unregistered [ 1099.522937] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1099.771875] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1099.777985] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1099.787529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1102.626812] Key type ._llcrypt registered [ 1102.628807] Key type .llcrypt registered [ 1102.827702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1102.873478] alg: No test for adler32 (adler32-zlib) [ 1103.861880] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1111.217253] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 23:25:15 (1788665115) [ 1112.883842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1115.180051] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 23:25:19 (1788665119) [ 1115.510358] LNet: 57119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.523702] LNetError: 57119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1115.547559] LNet: Removed LNI 10.1.2.3@tcp [ 1116.104625] Key type .llcrypt unregistered [ 1116.109500] Key type ._llcrypt unregistered [ 1123.272734] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1123.475473] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1123.479410] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1123.484679] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1125.178211] Key type ._llcrypt registered [ 1125.181439] Key type .llcrypt registered [ 1125.418440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1125.432991] alg: No test for adler32 (adler32-zlib) [ 1126.337971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1126.393953] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1134.581656] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 23:25:39 (1788665139) [ 1135.193621] LNet: 58337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.215813] LNetError: 58337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1135.240326] LNet: Removed LNI 10.1.2.3@tcp [ 1135.808417] Key type .llcrypt unregistered [ 1135.810183] Key type ._llcrypt unregistered [ 1142.532115] Key type ._llcrypt registered [ 1142.533762] Key type .llcrypt registered [ 1142.893831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1142.932130] alg: No test for adler32 (adler32-zlib) [ 1143.868235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1143.887928] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1151.949274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.121@tcp [ 1152.623348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_interrupt -r 1 [ 1152.681324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_interrupt -r 1 [ 1152.732777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1152.763892] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1153.013979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1153.039379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1153.083713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1153.120685] LNet: 1 local NIs in recovery (showing 1): 192.168.204.21@tcp [ 1153.133526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1153.171706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1153.223241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1153.274926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1153.323880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1153.382397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1153.403804] LNet: 58773:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1153.432490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1153.497818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1153.532788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1153.566701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1153.603389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1153.654492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1153.705376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1154.312398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_dropped -r 1 [ 1154.353796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_dropped -r 1 [ 1154.401383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1154.428852] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1154.437434] LNet: Skipped 1 previous similar message [ 1154.650493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.713519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.757574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.796779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.826806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1154.876345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.919500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.957800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.991297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1155.020388] LNet: 58885:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1155.065447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.118462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.185558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.220270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.298079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.363441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.428933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1156.042641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_aborted -r 1 [ 1156.077712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_aborted -r 1 [ 1156.130444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1156.149965] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1156.372410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1156.406753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1156.450043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1156.503222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1156.535321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1156.579694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1156.630305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1156.663190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1156.708294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1156.750390] LNet: 58996:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1156.774877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1156.833054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1156.852403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1156.882605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1156.923831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.002738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.075473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1157.650338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_no_route -r 1 [ 1157.694416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_no_route -r 1 [ 1157.738527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1158.059253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1158.101950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1158.143839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1158.203921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1158.267291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1158.329741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1158.383633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1158.421399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1158.485461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1158.534963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1158.606810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1158.661274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1158.691627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1158.757269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1158.802276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1158.845120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1159.470547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_timeout -r 1 [ 1159.508939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_timeout -r 1 [ 1159.555716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1159.584882] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1159.592917] LNet: Skipped 2 previous similar messages [ 1159.919342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1159.954946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1159.995160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1160.036862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1160.076057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1160.141793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1160.185800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1160.221954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1160.248671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.269693] LNet: 59218:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1160.286500] LNet: 59218:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1160.314569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.361957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.397896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.424729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.454481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1160.497272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1160.530622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1161.108222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_error -r 1 [ 1161.145585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_error -r 1 [ 1161.189216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1161.485665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.507818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1161.530396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.548342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.573640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1161.602919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.644296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.675792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.717158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.768854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1161.821144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1161.867102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.894932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1161.935805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1161.986649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.042264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1162.373420] LNet: 59371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1162.378898] LNetError: 59371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1163.430616] LNet: Removed LNI 192.168.204.21@tcp [ 1163.868665] Key type .llcrypt unregistered [ 1163.871273] Key type ._llcrypt unregistered [ 1186.501243] Key type ._llcrypt registered [ 1186.504057] Key type .llcrypt registered [ 1187.951945] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 23:26:32 (1788665192) [ 1189.347380] Key type .llcrypt unregistered [ 1189.349066] Key type ._llcrypt unregistered [ 1198.034173] Key type ._llcrypt registered [ 1198.038187] Key type .llcrypt registered [ 1198.314574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1198.331935] alg: No test for adler32 (adler32-zlib) [ 1199.312413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1199.336644] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1208.347220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.121@tcp [ 1219.738375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1219.826313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1219.856625] LNet: Added LNI 192.168.204.21@tcp1 [8/256/0/180] [ 1221.855269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_interrupt -r 1 [ 1221.896523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e local_interrupt -r 1 [ 1221.981620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_interrupt -r 1 [ 1222.076174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e local_interrupt -r 1 [ 1222.159773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e local_interrupt -r 1 [ 1222.209723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e local_interrupt -r 1 [ 1222.255517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e local_interrupt -r 1 [ 1222.318158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e local_interrupt -r 1 [ 1222.390350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1222.449217] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1222.462306] LNet: 1 local NIs in recovery (showing 1): 192.168.204.21@tcp [ 1222.851313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1222.882705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1222.923189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1223.002427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1223.057962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1223.123803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1223.190452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1223.256297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1223.301905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1223.323984] LNet: 60837:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1223.360563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1223.422781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1223.486580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1223.536720] LNet: There was an unexpected network error while writing to 192.168.204.21: rc = -22 [ 1223.543600] LNet: Skipped 2 previous similar messages [ 1223.565340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1223.621510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1223.706699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1223.768312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1224.770495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_dropped -r 1 [ 1224.851697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e local_dropped -r 1 [ 1224.932491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_dropped -r 1 [ 1224.982252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e local_dropped -r 1 [ 1225.039365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e local_dropped -r 1 [ 1225.080901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e local_dropped -r 1 [ 1225.136874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e local_dropped -r 1 [ 1225.174744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e local_dropped -r 1 [ 1225.224382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1225.268319] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1225.575445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1225.615453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1225.659701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1225.714670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1225.772988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1225.846813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1225.893621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1225.953681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1226.013090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.041352] LNet: 60960:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.090483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.144579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.180282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.242591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.297666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.400192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.487546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1227.334540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_aborted -r 1 [ 1227.397751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e local_aborted -r 1 [ 1227.493098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_aborted -r 1 [ 1227.572497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e local_aborted -r 1 [ 1227.626327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e local_aborted -r 1 [ 1227.689387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e local_aborted -r 1 [ 1227.751922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e local_aborted -r 1 [ 1227.795087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e local_aborted -r 1 [ 1227.847977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1227.879075] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1227.884233] LNet: Skipped 2 previous similar messages [ 1228.163898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1228.198598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1228.225774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.266095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.320215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1228.404425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.462192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.505133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.530943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.559728] LNet: 61083:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1228.608441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1228.639611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1228.681969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.713324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.754859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.782261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1228.818644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1229.447904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_no_route -r 1 [ 1229.489866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e local_no_route -r 1 [ 1229.536905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_no_route -r 1 [ 1229.581599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e local_no_route -r 1 [ 1229.627940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e local_no_route -r 1 [ 1229.669435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e local_no_route -r 1 [ 1229.730667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e local_no_route -r 1 [ 1229.796463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e local_no_route -r 1 [ 1229.855664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1230.143296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1230.179874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1230.206944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1230.231875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1230.275133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1230.308117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1230.341172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1230.390843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1230.427544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.462613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1230.491393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.532146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.578302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.609886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.659660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1230.718530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1231.243546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_timeout -r 1 [ 1231.276824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e local_timeout -r 1 [ 1231.305326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_timeout -r 1 [ 1231.337703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e local_timeout -r 1 [ 1231.381866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e local_timeout -r 1 [ 1231.448631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e local_timeout -r 1 [ 1231.518795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e local_timeout -r 1 [ 1231.566780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e local_timeout -r 1 [ 1231.636500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1231.941239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1232.012391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1232.063126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1232.117465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1232.151802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1232.199883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1232.243275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1232.313975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1232.371528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1232.395211] LNet: 61330:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1232.400784] LNet: 61330:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1232.456595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1232.506296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1232.554839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1232.594354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1232.641653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1232.673698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1232.707754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1233.313993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_error -r 1 [ 1233.353536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e local_error -r 1 [ 1233.395273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_error -r 1 [ 1233.433745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e local_error -r 1 [ 1233.465930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e local_error -r 1 [ 1233.511371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e local_error -r 1 [ 1233.565327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e local_error -r 1 [ 1233.643658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e local_error -r 1 [ 1233.724305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1233.770273] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1233.776243] LNet: Skipped 8 previous similar messages [ 1234.008783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1234.043600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1234.091731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1234.119615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1234.153352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1234.182775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1234.211787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1234.242700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1234.308823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1234.360478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.390469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.434785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.471825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1234.506562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1234.568036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1234.619140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1240.186515] LNet: 61529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.207985] LNetError: 61529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1240.227719] LNet: Removed LNI 192.168.204.21@tcp [ 1241.256912] LNet: Removed LNI 192.168.204.21@tcp1 [ 1241.759636] Key type .llcrypt unregistered [ 1241.763892] Key type ._llcrypt unregistered [ 1258.606642] Key type ._llcrypt registered [ 1258.609178] Key type .llcrypt registered [ 1259.572289] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 23:27:44 (1788665264) [ 1260.335482] Key type .llcrypt unregistered [ 1260.340869] Key type ._llcrypt unregistered [ 1265.865460] Key type ._llcrypt registered [ 1265.871221] Key type .llcrypt registered [ 1266.044939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1266.063766] alg: No test for adler32 (adler32-zlib) [ 1266.989177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1267.025659] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1274.141461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.121@tcp [ 1274.881443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e remote_dropped -r 1 [ 1274.909924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e remote_dropped -r 1 [ 1274.941610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1274.991259] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1275.253723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1275.304600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1275.339883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1275.374435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1275.404984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1275.451858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1275.506090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1275.546209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1275.581077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1275.620701] LNet: 62775:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1275.646398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1275.681599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1275.721229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1275.759578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1275.811202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1275.858495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1275.897209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1276.373429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e remote_error -r 1 [ 1276.404079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e remote_error -r 1 [ 1276.437130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1276.459451] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1276.674132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1276.701631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1276.740202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1276.772411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1276.801402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1276.835790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1276.866722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1276.897503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1276.943686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1276.965756] LNet: 62887:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1276.987023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1277.022717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1277.051397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1277.076741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1277.113386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1277.159366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1277.196212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1277.687880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e remote_timeout -r 1 [ 1277.719379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e remote_timeout -r 1 [ 1277.744688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1277.764472] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1277.980360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.007843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1278.033502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1278.064366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.094355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1278.127636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.168050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.198757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.229843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.251647] LNet: 62998:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.283087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.325621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.352901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.393892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.441277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.498786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.542386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1278.771896] LNet: 63039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.776375] LNetError: 63039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1278.795200] LNet: Removed LNI 192.168.204.21@tcp [ 1279.213079] Key type .llcrypt unregistered [ 1279.216904] Key type ._llcrypt unregistered [ 1295.745451] Key type ._llcrypt registered [ 1295.757524] Key type .llcrypt registered [ 1296.954821] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 23:28:21 (1788665301) [ 1298.059578] Key type .llcrypt unregistered [ 1298.062473] Key type ._llcrypt unregistered [ 1304.243055] Key type ._llcrypt registered [ 1304.246151] Key type .llcrypt registered [ 1304.507900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1304.535579] alg: No test for adler32 (adler32-zlib) [ 1305.409570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1305.429567] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1315.006637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.121@tcp [ 1323.904932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1323.956564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1323.991941] LNet: Added LNI 192.168.204.21@tcp1 [8/256/0/180] [ 1325.562168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e remote_dropped -r 1 [ 1325.611718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e remote_dropped -r 1 [ 1325.649956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e remote_dropped -r 1 [ 1325.694107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e remote_dropped -r 1 [ 1325.728210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e remote_dropped -r 1 [ 1325.779399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e remote_dropped -r 1 [ 1325.847361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e remote_dropped -r 1 [ 1325.923317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e remote_dropped -r 1 [ 1325.983636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1326.029590] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1326.043573] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.121@tcp [ 1326.337388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1326.385928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1326.423474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1326.458849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1326.520501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1326.586213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1326.672945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1326.716805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1326.765498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1326.792106] LNet: 64502:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1326.810928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1326.842252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1326.880942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1326.945121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1327.001227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1327.083091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1327.125072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1327.167637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1327.887892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e remote_error -r 1 [ 1327.932757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e remote_error -r 1 [ 1327.983987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e remote_error -r 1 [ 1328.013622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e remote_error -r 1 [ 1328.052685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e remote_error -r 1 [ 1328.099481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e remote_error -r 1 [ 1328.161980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e remote_error -r 1 [ 1328.214704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e remote_error -r 1 [ 1328.263339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1328.301734] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1328.307270] LNet: Skipped 2 previous similar messages [ 1328.613723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1328.669760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1328.718841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1328.768753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1328.816353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1328.848705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1328.897311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1328.957220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1329.010849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1329.038349] LNet: 64627:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1329.069639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1329.110881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1329.143155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1329.189555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1329.253757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1329.306871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1329.357852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1329.403606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1330.074277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e remote_timeout -r 1 [ 1330.110993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e remote_timeout -r 1 [ 1330.142674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e remote_timeout -r 1 [ 1330.179697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e remote_timeout -r 1 [ 1330.223727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e remote_timeout -r 1 [ 1330.263635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e remote_timeout -r 1 [ 1330.302975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e remote_timeout -r 1 [ 1330.343762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e remote_timeout -r 1 [ 1330.379234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.121@tcp [ 1330.396348] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1330.403296] LNet: Skipped 1 previous similar message [ 1330.618875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1330.650860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1330.693282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1330.731633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1330.759526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1330.785328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1330.809707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1330.841151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1330.865596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1330.882309] LNet: 64752:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1330.895609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1330.925279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1330.955630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1330.995048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1331.029581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1331.060098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1331.098612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1331.142481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1337.742749] LNet: 64830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1337.747987] LNetError: 64830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1337.755269] LNet: Removed LNI 192.168.204.21@tcp [ 1337.780818] LNet: Removed LNI 192.168.204.21@tcp1 [ 1338.248760] Key type .llcrypt unregistered [ 1338.254124] Key type ._llcrypt unregistered [ 1357.239398] Key type ._llcrypt registered [ 1357.243688] Key type .llcrypt registered [ 1358.678780] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 23:29:23 (1788665363) [ 1360.246822] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1362.107810] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 23:29:26 (1788665366) [ 1362.991467] Key type .llcrypt unregistered [ 1362.996107] Key type ._llcrypt unregistered [ 1369.497318] Key type ._llcrypt registered [ 1369.499370] Key type .llcrypt registered [ 1369.671677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1369.692168] alg: No test for adler32 (adler32-zlib) [ 1370.649147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1370.677694] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1377.430934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.121@tcp [ 1378.165505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e network_timeout -r 1 [ 1378.212831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e network_timeout -r 1 [ 1378.262927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.121@tcp [ 1378.278772] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1378.485424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1378.514948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1378.549243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1378.585755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1378.617561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1378.647515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1378.688513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1378.712134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1378.741963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1378.758032] LNet: 66371:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1378.779556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1378.828181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1378.884949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1378.911133] LNet: 1 local NIs in recovery (showing 1): 192.168.204.21@tcp [ 1378.956175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1379.007767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1379.059917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1379.386313] LNet: 66411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1379.392057] LNetError: 66411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1379.405761] LNet: Removed LNI 192.168.204.21@tcp [ 1379.737451] Key type .llcrypt unregistered [ 1379.739226] Key type ._llcrypt unregistered [ 1393.987639] Key type ._llcrypt registered [ 1393.989609] Key type .llcrypt registered [ 1394.146265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1394.158797] alg: No test for adler32 (adler32-zlib) [ 1395.057072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1395.088577] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1400.731243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.121@tcp [ 1408.853289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1408.903798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1408.962945] LNet: Added LNI 192.168.204.21@tcp1 [8/256/0/180] [ 1410.651976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e network_timeout -r 1 [ 1410.689871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e network_timeout -r 1 [ 1410.721758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e network_timeout -r 1 [ 1410.762580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e network_timeout -r 1 [ 1410.806769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e network_timeout -r 1 [ 1410.839633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e network_timeout -r 1 [ 1410.875856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e network_timeout -r 1 [ 1410.914328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e network_timeout -r 1 [ 1410.955793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.121@tcp [ 1410.975742] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1411.213628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1411.244877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1411.276648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1411.303919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1411.331887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1411.363444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1411.391330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1411.420465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1411.453420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1411.469091] LNet: 67285:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1411.482815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1411.488519] LNet: 1 local NIs in recovery (showing 1): 192.168.204.21@tcp [ 1411.492229] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1411.498719] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.121@tcp [ 1411.518445] LNet: Skipped 1 previous similar message [ 1411.539433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1411.571395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1411.618895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1411.661260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1411.714996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1415.740691] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1416.063273] LNet: 67359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1416.069547] LNetError: 67359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1416.079214] LNet: Removed LNI 192.168.204.21@tcp [ 1416.103186] LNet: Removed LNI 192.168.204.21@tcp1 [ 1416.435495] Key type .llcrypt unregistered [ 1416.440136] Key type ._llcrypt unregistered [ 1430.456827] Key type ._llcrypt registered [ 1430.459697] Key type .llcrypt registered [ 1431.860784] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 23:30:36 (1788665436) [ 1432.132898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1432.146178] alg: No test for adler32 (adler32-zlib) [ 1433.017932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1433.052208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1433.071181] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1433.656694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1433.671614] LNet: Added LNI 192.168.204.21@tcp1 [8/256/0/180] [ 1434.438325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 1434.500117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1434.540906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1434.576704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1434.613363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1434.651495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1434.694481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.21@tcp [ 1434.713249] LNetError: 68455:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.21@tcp to recovery [ 1435.103392] LNet: 1 local NIs in recovery (showing 1): 192.168.204.21@tcp [ 1435.103947] LNet: There was an unexpected network error while writing to 192.168.204.21: rc = -22 [ 1437.151876] LNet: There was an unexpected network error while writing to 192.168.204.21: rc = -22 [ 1438.109380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1441.247401] LNet: There was an unexpected network error while writing to 192.168.204.21: rc = -22 [ 1441.613327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1441.650949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1441.726178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1441.758432] LNet: 68562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1441.767560] LNetError: 68562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1442.783367] LNet: Removed LNI 192.168.204.21@tcp [ 1442.788587] LNetError: 68562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1442.810630] LNet: Removed LNI 192.168.204.21@tcp1 [ 1442.859219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1442.906317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1442.942683] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1443.561265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1443.583093] LNet: Added LNI 192.168.204.21@tcp1 [8/256/0/180] [ 1444.449796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 1444.485842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1444.552564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1444.652849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1444.712595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1444.769348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1444.802642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1444.840259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.21@tcp [ 1444.862818] LNetError: 68835:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.21@tcp to recovery [ 1444.972150] LNet: There was an unexpected network error while writing to 192.168.204.21: rc = -22 [ 1447.273544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1451.103942] LNet: There was an unexpected network error while writing to 192.168.204.21: rc = -22 [ 1451.109755] LNet: Skipped 1 previous similar message [ 1456.244530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1456.286461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1456.351373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1463.461073] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 23:31:08 (1788665468) [ 1463.638313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1463.669070] LNet: 69553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.674727] LNetError: 69553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.681083] LNet: Removed LNI 192.168.204.21@tcp [ 1463.693346] LNet: Removed LNI 192.168.204.21@tcp1 [ 1463.706742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1463.755526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1463.778970] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1464.421769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1464.446074] LNet: Added LNI 192.168.204.21@tcp1 [8/256/0/180] [ 1465.146574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 1465.200911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1465.248116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1465.286545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1465.319293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1465.356810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1465.386865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.21@tcp --health 0 [ 1465.477180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1465.760234] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.21@tcp [ 1466.784294] LNet: There was an unexpected network error while writing to 192.168.204.21: rc = -22 [ 1466.790565] LNet: Skipped 1 previous similar message [ 1480.696574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1480.824400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1480.858197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1480.889559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.21@tcp --health 500 [ 1480.980917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1486.636805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1486.715453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1486.739721] LNet: 70037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1486.749417] LNetError: 70037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.755401] LNetError: 70037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1486.758639] LNet: Removed LNI 192.168.204.21@tcp [ 1486.771831] LNet: Removed LNI 192.168.204.21@tcp1 [ 1486.790141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.831975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1486.857817] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1487.674679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1487.707920] LNet: Added LNI 192.168.204.21@tcp1 [8/256/0/180] [ 1488.612683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 1488.647691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1488.700565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1488.754265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1488.794940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1488.841906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1488.890861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1488.940460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.21@tcp --health 0 [ 1489.888466] LNet: There was an unexpected network error while writing to 192.168.204.21: rc = -22 [ 1489.901420] LNet: Skipped 3 previous similar messages [ 1490.269602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1499.451490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1499.500083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1499.544352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1506.838537] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 23:31:51 (1788665511) [ 1507.332434] LNet: 71073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1507.336893] LNetError: 71073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1507.342831] LNetError: 71073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1507.349266] LNet: Removed LNI 192.168.204.21@tcp [ 1507.362401] LNet: Removed LNI 192.168.204.21@tcp1 [ 1507.974224] Key type .llcrypt unregistered [ 1507.977279] Key type ._llcrypt unregistered [ 1514.039833] Key type ._llcrypt registered [ 1514.042271] Key type .llcrypt registered [ 1514.356524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1514.372581] alg: No test for adler32 (adler32-zlib) [ 1515.342445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1515.381715] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1524.312754] Lustre: DEBUG MARKER: Initial discovery [ 1524.362939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.121@tcp [ 1526.722719] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1526.847658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1526.925624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.121@tcp [ 1526.973319] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1527.002638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1527.046902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.121@tcp -r 1 -m GET -l 3 [ 1527.102250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.121@tcp -r 1 -m GET -e local_error [ 1527.169836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.121@tcp -r 1 -m PUT -l 6 [ 1527.711467] LNet: 1 local NIs in recovery (showing 1): 192.168.204.21@tcp [ 1529.114175] Lustre: DEBUG MARKER: Force oleg421-client.virtnet to discover 192.168.204.121@tcp (in background) [ 1529.163699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.121@tcp [ 1530.507462] Lustre: DEBUG MARKER: Force oleg421-server to discover 192.168.204.21@tcp [ 1532.256293] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1533.190409] Lustre: DEBUG MARKER: Wait for 71824 [ 1535.026284] Lustre: DEBUG MARKER: Finished wait on 71824 [ 1535.076969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1535.119360] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1535.137141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1535.461534] LNet: 72281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.472471] LNetError: 72281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.490412] LNet: Removed LNI 192.168.204.21@tcp [ 1535.795533] Key type .llcrypt unregistered [ 1535.798152] Key type ._llcrypt unregistered [ 1550.564809] Key type ._llcrypt registered [ 1550.566348] Key type .llcrypt registered [ 1551.988676] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 23:32:36 (1788665556) [ 1552.487485] Key type .llcrypt unregistered [ 1552.489221] Key type ._llcrypt unregistered [ 1558.530558] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1558.625345] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1558.633628] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1558.642755] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1561.251635] Key type ._llcrypt registered [ 1561.253347] Key type .llcrypt registered [ 1561.451208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1561.471198] alg: No test for adler32 (adler32-zlib) [ 1562.423916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.529340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1562.567938] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1563.282432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1563.298820] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1571.247958] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 23:32:56 (1788665576) [ 1571.512787] LNet: 74368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.526862] LNetError: 74368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1571.532639] LNet: Removed LNI 10.1.2.3@tcp [ 1571.553508] LNet: Removed LNI 192.168.204.21@tcp [ 1572.262982] Key type .llcrypt unregistered [ 1572.267397] Key type ._llcrypt unregistered [ 1579.546708] Key type ._llcrypt registered [ 1579.550808] Key type .llcrypt registered [ 1579.788031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1579.811809] alg: No test for adler32 (adler32-zlib) [ 1580.872739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.940642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1580.961108] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1581.761365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1581.785779] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1582.877407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.21@tcp [ 1583.967164] LNetError: Unexpected error -22 connecting to 192.168.204.21@tcp at host 192.168.204.21:988 [ 1589.252567] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 23:33:14 (1788665594) [ 1589.366324] LNet: 75489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.380915] LNetError: 75489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1589.386907] LNet: Removed LNI 10.1.2.3@tcp [ 1589.418494] LNet: Removed LNI 192.168.204.21@tcp [ 1590.156472] Key type .llcrypt unregistered [ 1590.159202] Key type ._llcrypt unregistered [ 1596.350116] Key type ._llcrypt registered [ 1596.352144] Key type .llcrypt registered [ 1596.596575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1596.623660] alg: No test for adler32 (adler32-zlib) [ 1597.551901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.591936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1597.629697] LNet: Added LNI 192.168.204.21@tcp1 [8/256/0/180] [ 1598.464450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1598.488557] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 1599.428507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.21@tcp1 --nid 192.168.204.21@tcp2 [ 1603.891559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.21@tcp1 192.168.204.21@tcp2 [ 1603.919676] LNetError: 76329:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.21@tcp1 to 192.168.204.21@tcp2 - Route table may be misconfigured [ 1603.938221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.21@tcp2 192.168.204.21@tcp1 [ 1611.126143] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 23:33:35 (1788665615) [ 1611.232321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.276654] LNet: 76894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.290937] LNetError: 76894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.319923] LNet: Removed LNI 192.168.204.21@tcp1 [ 1612.341909] LNetError: 76894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.362198] LNet: Removed LNI 192.168.204.21@tcp2 [ 1612.440621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.562511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1612.621152] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1613.406240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1613.446862] LNet: Added LNI 192.168.204.21@tcp1 [8/256/0/180] [ 1614.547183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 1614.604753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.21@tcp 192.168.204.21@tcp [ 1614.652361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.21@tcp1 192.168.204.21@tcp1 [ 1614.711482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.21@tcp -d 192.168.204.21@tcp -e network_timeout [ 1614.756482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -e network_timeout [ 1614.813342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -e network_timeout [ 1614.871268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -e network_timeout [ 1614.925641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 1614.958244] LNet: There was an unexpected network error while writing to 192.168.204.21: rc = -22 [ 1614.982189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1615.126827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1615.262469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1615.583237] LNet: 1 local NIs in recovery (showing 1): 192.168.204.21@tcp [ 1622.412229] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 23:33:47 (1788665627) [ 1622.635807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.661552] LNet: 77763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.667806] LNetError: 77763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.672246] LNet: Removed LNI 192.168.204.21@tcp [ 1623.719168] LNet: Removed LNI 192.168.204.21@tcp1 [ 1623.787826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.925633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1624.237695] LNet: 77801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.828565] Key type .llcrypt unregistered [ 1624.837601] Key type ._llcrypt unregistered [ 1638.494909] Key type ._llcrypt registered [ 1638.497817] Key type .llcrypt registered [ 1640.073896] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 23:34:04 (1788665644) [ 1640.589595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1640.610306] alg: No test for adler32 (adler32-zlib) [ 1641.660941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.645944] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1645.914741] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 23:34:10 (1788665650) [ 1646.041382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.084942] LNet: 78906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.117251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.199197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1646.232482] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1646.990656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1647.024360] LNet: Added LNI 192.168.204.21@tcp1 [8/256/0/180] [ 1648.126847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 1648.183951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp1 [ 1648.251547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp1 [ 1655.676945] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 23:34:20 (1788665660) [ 1657.565922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg421-server [ 1659.800784] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 23:34:24 (1788665664) [ 1661.424307] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg421-server [ 1663.241370] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 23:34:28 (1788665668) [ 1665.091722] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg421-server [ 1667.332452] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 23:34:31 (1788665671) [ 1669.048659] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg421-server [ 1670.794730] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 23:34:35 (1788665675) [ 1672.677499] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg421-server [ 1674.347199] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 23:34:39 (1788665679) [ 1676.257220] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg421-server [ 1678.237357] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 23:34:42 (1788665682) [ 1680.774380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg421-server [ 1682.998809] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 23:34:47 (1788665687) [ 1685.059172] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg421-server [ 1687.124748] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 23:34:51 (1788665691) [ 1688.954832] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg421-server [ 1690.874410] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 23:34:55 (1788665695) [ 1691.216385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.258799] LNet: 82790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.271419] LNetError: 82790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.283866] LNet: Removed LNI 192.168.204.21@tcp [ 1692.326992] LNet: Removed LNI 192.168.204.21@tcp1 [ 1692.384092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.511488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1692.538710] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1693.354748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1693.420557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 1693.591555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1693.616292] LNet: 82938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1693.624559] LNetError: 82938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1693.631445] LNetError: 82938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1693.651533] LNet: Removed LNI 192.168.204.21@tcp [ 1693.696424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.777317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1693.807225] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1694.502523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 1694.719430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.746679] LNetError: 83083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.763943] LNet: Removed LNI 192.168.204.21@tcp [ 1694.807608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.885711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1694.926977] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1695.741551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1695.827455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 1696.066373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.094722] LNet: 83230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.103058] LNet: 83230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1697.127479] LNet: Removed LNI 192.168.204.21@tcp [ 1697.159354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.215660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1697.255218] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1698.213827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 1698.482198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.527920] LNetError: 83376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1698.537388] LNetError: 83376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1698.580354] LNet: Removed LNI 192.168.204.21@tcp [ 1698.637816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.745056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1698.772744] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1699.631183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1699.712287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 1699.988333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1701.097441] LNet: Removed LNI 192.168.204.21@tcp [ 1701.155790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.229597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1701.260702] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1702.139771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 1702.417082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.472678] LNet: 83669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.479686] LNet: 83669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1703.529297] LNet: Removed LNI 192.168.204.21@tcp [ 1703.567760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.662081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.696251] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1704.509728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1704.607869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 1705.972518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.995976] LNetError: 83825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.002931] LNetError: 83825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1706.019351] LNet: Removed LNI 192.168.204.21@tcp [ 1706.071722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.149152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1706.214602] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1706.928921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 1708.361907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.419464] LNet: Removed LNI 192.168.204.21@tcp [ 1709.471420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.525146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1709.547665] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1710.550674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1710.646752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 1710.958940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.988435] LNet: 84127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.996596] LNet: 84127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1712.042893] LNet: Removed LNI 192.168.204.21@tcp [ 1712.094101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.178559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1712.205562] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1713.128419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 1714.694454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.730959] LNetError: 84282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1714.745478] LNetError: 84282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1714.773384] LNet: Removed LNI 192.168.204.21@tcp [ 1714.833574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.896707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1714.916810] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1715.818260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1715.926957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 1716.209705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1717.285742] LNet: Removed LNI 192.168.204.21@tcp [ 1717.314908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.404531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1717.430696] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1718.192665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 1718.539911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.589650] LNet: Removed LNI 192.168.204.21@tcp [ 1719.622196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.681561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1719.704982] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1720.433151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1720.473168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.498243] LNet: Removed LNI 192.168.204.21@tcp [ 1720.512456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.560411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1720.576820] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1721.195623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1721.276291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.324659] LNet: Removed LNI 192.168.204.21@tcp [ 1721.338982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.390511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1721.496747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.525816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.574639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1721.647993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.680206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.725751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1721.800605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.848740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.927580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1721.961137] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1722.644610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.680308] LNet: Removed LNI 192.168.204.21@tcp [ 1722.696754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.747178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1730.075867] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 23:35:34 (1788665734) [ 1730.439851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.466711] LNet: 85596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.476763] LNet: 85596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1730.494227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.537813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1730.555464] LNet: Added LNI 192.168.204.21@tcp231 [8/256/0/180] [ 1731.546552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.567876] LNetError: 85734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.580366] LNetError: 85734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1731.594033] LNet: Removed LNI 192.168.204.21@tcp231 [ 1731.615580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.701591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1731.731575] LNet: Added LNI 192.168.204.21@tcp231 [8/256/0/180] [ 1731.741355] LNet: 85742:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1732.176968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1732.200372] LNet: Removed LNI 192.168.204.21@tcp231 [ 1732.221306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1732.255798] LNet: Added LNI 192.168.204.21@tcp231 [8/256/0/180] [ 1733.919938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.957368] LNet: Removed LNI 192.168.204.21@tcp231 [ 1733.973937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.045407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1734.075886] LNet: Added LNI 192.168.204.21@tcp231 [8/256/0/0] [ 1734.083499] LNet: 85895:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1734.414695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1734.453202] LNet: Removed LNI 192.168.204.21@tcp231 [ 1734.483879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1734.519291] LNet: Added LNI 192.168.204.21@tcp231 [8/256/0/0] [ 1735.552404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.583972] LNet: Removed LNI 192.168.204.21@tcp231 [ 1735.598767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.657416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1735.676322] LNet: Added LNI 192.168.204.21@tcp231 [8/256/0/60] [ 1735.680634] LNet: 86046:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1735.905809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1735.943599] LNet: Removed LNI 192.168.204.21@tcp231 [ 1735.971696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1736.000527] LNet: Added LNI 192.168.204.21@tcp231 [8/256/0/60] [ 1737.027621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.070563] LNet: Removed LNI 192.168.204.21@tcp231 [ 1737.087469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.151323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1737.192612] LNet: Added LNI 192.168.204.21@tcp231 [8/256/0/180] [ 1737.838184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1737.874707] LNet: Removed LNI 192.168.204.21@tcp231 [ 1737.899941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1737.989644] LNet: Added LNI 192.168.204.21@tcp231 [8/256/0/180] [ 1746.913377] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 23:35:51 (1788665751) [ 1746.976430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1747.055924] LNet: Removed LNI 192.168.204.21@tcp231 [ 1747.096335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.217809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1747.261975] LNet: Added LNI 192.168.204.21@tcp232 [8/256/0/180] [ 1748.130570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1748.192953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1748.261490] LNet: Removed LNI 192.168.204.21@tcp232 [ 1748.288544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1748.360415] LNet: Added LNI 192.168.204.21@tcp232 [8/256/0/180] [ 1748.896497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1748.943412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1748.990561] LNet: Removed LNI 192.168.204.21@tcp232 [ 1749.021174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1749.077164] LNet: Added LNI 192.168.204.21@tcp232 [8/256/0/180] [ 1749.794550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1749.821797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1749.850703] LNet: Removed LNI 192.168.204.21@tcp232 [ 1749.876837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1749.919427] LNet: Added LNI 192.168.204.21@tcp232 [8/256/0/180] [ 1750.665298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1750.728972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1750.781068] LNet: Removed LNI 192.168.204.21@tcp232 [ 1750.811430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1758.668936] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 23:36:02 (1788665762) [ 1760.283754] Key type .llcrypt unregistered [ 1760.286083] Key type ._llcrypt unregistered [ 1767.654536] Key type ._llcrypt registered [ 1767.656159] Key type .llcrypt registered [ 1768.112463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1768.127844] alg: No test for adler32 (adler32-zlib) [ 1769.056721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1769.082235] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1779.064087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.121@tcp [ 1789.653453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1789.726164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1789.762278] LNet: Added LNI 192.168.204.21@tcp1 [8/256/0/180] [ 1791.188084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_interrupt -r 2 [ 1791.242668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e local_interrupt -r 2 [ 1791.303594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_interrupt -r 2 [ 1791.364780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e local_interrupt -r 2 [ 1791.407698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e local_interrupt -r 2 [ 1791.445284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e local_interrupt -r 2 [ 1791.486386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e local_interrupt -r 2 [ 1791.547755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e local_interrupt -r 2 [ 1791.597146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.21@tcp 192.168.204.121@tcp [ 1791.609351] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1791.613100] LNet: 1 local NIs in recovery (showing 1): 192.168.204.21@tcp [ 1791.834420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1791.866965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1791.891332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1791.918713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1791.956365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1791.982764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1792.023088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1792.090667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1792.168148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1792.201699] LNet: 88701:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1792.268229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1792.322757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1792.364484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1792.396596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1792.438894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1792.494272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1793.085617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_dropped -r 2 [ 1793.151576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e local_dropped -r 2 [ 1793.198639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_dropped -r 2 [ 1793.263716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e local_dropped -r 2 [ 1793.322628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e local_dropped -r 2 [ 1793.356351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e local_dropped -r 2 [ 1793.436394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e local_dropped -r 2 [ 1793.517231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e local_dropped -r 2 [ 1793.570521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.21@tcp 192.168.204.121@tcp [ 1793.591620] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1793.873680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1793.919038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1793.955393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1794.007047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1794.056564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1794.135436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1794.182272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1794.218936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1794.258910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1794.290966] LNet: 88823:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1794.318243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1794.372427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1794.404236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1794.454090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1794.496476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1794.535452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1795.193764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_aborted -r 2 [ 1795.241844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e local_aborted -r 2 [ 1795.292370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_aborted -r 2 [ 1795.343560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e local_aborted -r 2 [ 1795.391927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e local_aborted -r 2 [ 1795.439320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e local_aborted -r 2 [ 1795.494205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e local_aborted -r 2 [ 1795.552552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e local_aborted -r 2 [ 1795.603630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.21@tcp 192.168.204.121@tcp [ 1795.650814] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1796.007836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1796.047897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1796.080153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1796.124927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1796.153527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1796.211505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1796.255730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1796.314465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1796.377902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1796.406990] LNet: 88946:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1796.433394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1796.463392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1796.505466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1796.533585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1796.556246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1796.584652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1797.219363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_no_route -r 2 [ 1797.276856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e local_no_route -r 2 [ 1797.345449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_no_route -r 2 [ 1797.409448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e local_no_route -r 2 [ 1797.473646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e local_no_route -r 2 [ 1797.542210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e local_no_route -r 2 [ 1797.611603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e local_no_route -r 2 [ 1797.681724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e local_no_route -r 2 [ 1797.742281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.21@tcp 192.168.204.121@tcp [ 1797.765124] LNet: There was an unexpected network error while writing to 192.168.204.121: rc = -22 [ 1798.016234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1798.069621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1798.107519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1798.155284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1798.208679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1798.252294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1798.283690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1798.321314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1798.362646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1798.417714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1798.467872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1798.533198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1798.571267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1798.600572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1798.647284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1799.167820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e local_timeout -r 2 [ 1799.228453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e local_timeout -r 2 [ 1799.294445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e local_timeout -r 2 [ 1799.335962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e local_timeout -r 2 [ 1799.399566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e local_timeout -r 2 [ 1799.461175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e local_timeout -r 2 [ 1799.514878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e local_timeout -r 2 [ 1799.570591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e local_timeout -r 2 [ 1799.629829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.21@tcp 192.168.204.121@tcp [ 1799.918335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1799.979192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1800.039317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1800.077606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1800.134939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1800.186919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1800.236861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1800.292945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1800.325249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1800.362244] LNet: 89191:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1800.376652] LNet: 89191:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1800.395920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1800.430317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1800.462727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1800.501678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1800.549245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1800.588320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1801.087538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp -m GET -e remote_dropped -r 2 [ 1801.124813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.121@tcp1 -m GET -e remote_dropped -r 2 [ 1801.163619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp -m GET -e remote_dropped -r 2 [ 1801.213463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp -d 192.168.204.21@tcp1 -m GET -e remote_dropped -r 2 [ 1801.256439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp -m GET -e remote_dropped -r 2 [ 1801.305665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.121@tcp1 -m GET -e remote_dropped -r 2 [ 1801.342144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp -m GET -e remote_dropped -r 2 [ 1801.407267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.21@tcp1 -d 192.168.204.21@tcp1 -m GET -e remote_dropped -r 2 [ 1801.453180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.21@tcp 192.168.204.121@tcp [ 1801.480784] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.121@tcp [ 1801.694440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1801.742232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1801.787765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1801.833359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1801.879658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1801.915927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1801.951668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1801.992802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1802.018121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1802.052391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1802.096443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1802.134749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1802.167594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1802.198211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1802.232830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1807.593445] LNet: 89387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.601720] LNetError: 89387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.609384] LNet: Removed LNI 192.168.204.21@tcp [ 1807.623869] LNet: Removed LNI 192.168.204.21@tcp1 [ 1808.046532] Key type .llcrypt unregistered [ 1808.053727] Key type ._llcrypt unregistered [ 1824.343589] Key type ._llcrypt registered [ 1824.347474] Key type .llcrypt registered [ 1825.779471] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 23:37:10 (1788665830) [ 1826.236913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1826.249769] alg: No test for adler32 (adler32-zlib) [ 1827.131263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.176806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.195460] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1827.864644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1827.903922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.21@tcp [ 1827.923814] LNetError: 90365:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.21@tcp to recovery [ 1828.004424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.21@tcp [ 1828.129081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1835.952583] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 23:37:20 (1788665840) [ 1836.099371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1836.136583] LNet: 90949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.142146] LNetError: 90949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1836.161888] LNet: Removed LNI 192.168.204.21@tcp [ 1836.190151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.264117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1836.310896] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1837.143949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1837.188577] LNet: Added LNI 192.168.204.21@tcp1 [8/256/0/180] [ 1838.486435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.21@tcp [ 1838.536027] LNetError: 91222:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.21@tcp to recovery [ 1839.327366] LNet: 1 local NIs in recovery (showing 1): 192.168.204.21@tcp [ 1839.330543] LNet: There was an unexpected network error while writing to 192.168.204.21: rc = -22 [ 1839.909777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.21@tcp [ 1848.991751] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 23:37:33 (1788665853) [ 1851.444665] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg421-server [ 1853.654927] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 23:37:38 (1788665858) [ 1855.586518] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg421-server [ 1857.171746] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 23:37:42 (1788665862) [ 1859.353988] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg421-server [ 1860.904087] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 23:37:45 (1788665865) [ 1861.221722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1861.240656] LNet: 92852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1861.258323] LNetError: 92852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1861.272243] LNet: Removed LNI 192.168.204.21@tcp [ 1861.285834] LNet: Removed LNI 192.168.204.21@tcp1 [ 1861.315665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1861.388299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1861.416611] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1862.202413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1862.283192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1862.331722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1862.468570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1862.510250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1862.596802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1868.940281] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 23:37:53 (1788665873) [ 1869.288291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1869.320742] LNet: 93583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.326280] LNetError: 93583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.330702] LNetError: 93583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1869.351961] LNet: Removed LNI 192.168.204.21@tcp [ 1869.379833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1869.454854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1869.485852] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1876.751618] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 23:38:01 (1788665881) [ 1877.127777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1877.176522] LNet: 94287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.185668] LNetError: 94287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.202550] LNet: Removed LNI 192.168.204.21@tcp [ 1877.246911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1877.318507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1877.342234] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1884.902514] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 23:38:09 (1788665889) [ 1886.775879] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1888.651954] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 23:38:13 (1788665893) [ 1889.094776] LNet: 95322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.105602] LNetError: 95322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.122562] LNet: Removed LNI 192.168.204.21@tcp [ 1889.875964] Key type .llcrypt unregistered [ 1889.882245] Key type ._llcrypt unregistered [ 1895.578669] Key type ._llcrypt registered [ 1895.584260] Key type .llcrypt registered [ 1895.830506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1895.860782] alg: No test for adler32 (adler32-zlib) [ 1896.872523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1896.902847] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1904.799086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.121@tcp [ 1910.560311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.121@tcp [ 1910.811425] LNet: 95737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.821947] LNetError: 95737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.837327] LNet: Removed LNI 192.168.204.21@tcp [ 1911.248512] Key type .llcrypt unregistered [ 1911.250177] Key type ._llcrypt unregistered [ 1923.900875] Key type ._llcrypt registered [ 1923.903276] Key type .llcrypt registered [ 1925.288794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1927.293710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1929.108459] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 23:38:53 (1788665933) [ 1931.097237] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1933.155615] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 23:38:57 (1788665937) [ 1935.048680] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg421-server [ 1936.831790] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 23:39:01 (1788665941) [ 1938.478337] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg421-server [ 1940.570478] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 23:39:05 (1788665945) [ 1942.159715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1942.168075] alg: No test for adler32 (adler32-zlib) [ 1943.203568] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 1943.469338] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 1945.295171] Key type lgssc registered [ 1946.646933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1969.267887] Key type lgssc unregistered [ 1969.537978] LNet: 98909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.542785] LNetError: 98909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1969.567412] LNet: Removed LNI 192.168.204.21@tcp [ 1970.473733] Key type .llcrypt unregistered [ 1970.476462] Key type ._llcrypt unregistered [ 1987.450138] Key type ._llcrypt registered [ 1987.451179] Key type .llcrypt registered [ 1989.273747] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 23:39:53 (1788665993) [ 1989.837945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1989.865809] alg: No test for adler32 (adler32-zlib) [ 1990.871064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1990.955659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1991.018754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1991.094640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1991.133067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1998.224886] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 23:40:02 (1788666002) [ 1998.486705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1998.547321] LNet: 100324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.613704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1998.712817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1998.749446] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2000.201319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2000.219592] LNet: 100471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.229442] LNetError: 100471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.243487] LNet: Removed LNI 192.168.204.21@tcp [ 2000.270453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2000.344275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2000.361858] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2000.914149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2000.942217] LNetError: 100499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.986699] LNet: Removed LNI 192.168.204.21@tcp [ 2001.025963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2001.149453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2001.192402] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2008.965589] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 23:40:13 (1788666013) [ 2009.299701] LNet: 101089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.309325] LNet: 101089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2009.317581] LNetError: 101089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.340279] LNet: Removed LNI 192.168.204.21@tcp [ 2010.072630] Key type .llcrypt unregistered [ 2010.075186] Key type ._llcrypt unregistered [ 2017.380328] Key type ._llcrypt registered [ 2017.384511] Key type .llcrypt registered [ 2017.545224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2017.558932] alg: No test for adler32 (adler32-zlib) [ 2018.479247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2018.560358] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2018.563558] LNet: *** cfs_fail_loc=e003, val=100*** [ 2019.263329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2019.276799] LNet: *** cfs_fail_loc=e003, val=100*** [ 2019.279468] LNet: Skipped 2 previous similar messages [ 2019.284167] Ping reply received with mapping: [ 2019.285786] found metadata reply NI[192.168.204.21@tcp] [ 2019.289892] 15 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2019.292969] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2019.295889] 5a 5a 5a 5a 5a 5a 5a 5a [ 2019.393934] LNet: 101484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.403465] LNetError: 101484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.428901] LNet: Removed LNI 192.168.204.21@tcp [ 2020.042358] Key type .llcrypt unregistered [ 2020.048601] Key type ._llcrypt unregistered [ 2032.579668] Key type ._llcrypt registered [ 2032.582571] Key type .llcrypt registered [ 2034.244522] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 23:40:38 (1788666038) [ 2034.993512] Key type .llcrypt unregistered [ 2034.996500] Key type ._llcrypt unregistered [ 2041.923494] Key type ._llcrypt registered [ 2041.928461] Key type .llcrypt registered [ 2042.120366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2042.132568] alg: No test for adler32 (adler32-zlib) [ 2043.304872] LNetError: 102521:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2043.836429] Key type .llcrypt unregistered [ 2043.841921] Key type ._llcrypt unregistered [ 2049.314540] Key type ._llcrypt registered [ 2049.317857] Key type .llcrypt registered [ 2050.433669] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 23:40:55 (1788666055) [ 2051.195634] Key type .llcrypt unregistered [ 2051.207258] Key type ._llcrypt unregistered [ 2058.187915] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2058.193489] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2060.903955] Key type ._llcrypt registered [ 2060.908448] Key type .llcrypt registered [ 2061.066082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2061.088183] alg: No test for adler32 (adler32-zlib) [ 2062.115510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2062.224370] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2074.613365] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 23:41:18 (1788666078) [ 2074.771168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2074.797656] LNet: 104289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2074.803867] LNetError: 104289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2074.849733] LNet: Removed LNI 10.1.2.3@tcp [ 2074.880617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2075.921717] Key type .llcrypt unregistered [ 2075.923933] Key type ._llcrypt unregistered [ 2083.957271] Key type ._llcrypt registered [ 2083.958868] Key type .llcrypt registered [ 2084.209607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2084.226922] alg: No test for adler32 (adler32-zlib) [ 2085.085428] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2085.130199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2085.245109] LNet: 104581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.252199] LNetError: 104581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.268659] LNet: Removed LNI 192.168.204.21@tcp [ 2086.132358] Key type .llcrypt unregistered [ 2086.137522] Key type ._llcrypt unregistered [ 2098.877628] Key type ._llcrypt registered [ 2098.880492] Key type .llcrypt registered [ 2100.075500] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 23:41:44 (1788666104) [ 2100.970346] Key type .llcrypt unregistered [ 2100.971930] Key type ._llcrypt unregistered [ 2109.907395] Key type ._llcrypt registered [ 2109.911518] Key type .llcrypt registered [ 2110.274445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2110.306332] alg: No test for adler32 (adler32-zlib) [ 2120.637439] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 23:42:05 (1788666125) [ 2120.989605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2121.052979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.186668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2121.246881] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2122.545957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2122.594417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2122.630252] LNet: Removed LNI 192.168.204.21@tcp [ 2122.973947] LNet: 106324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.502382] Key type .llcrypt unregistered [ 2123.508033] Key type ._llcrypt unregistered [ 2138.179140] Key type ._llcrypt registered [ 2138.180966] Key type .llcrypt registered [ 2139.774832] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 23:42:24 (1788666144) [ 2140.123722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2140.157498] alg: No test for adler32 (adler32-zlib) [ 2141.245620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2141.330759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2141.363496] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2142.021598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2142.072371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.21@tcp [ 2142.096436] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2142.102046] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2142.108930] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2142, 0, 0 [ 2142.113772] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2142.115012] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2142.119780] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2142.122131] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2142, 0, 0 [ 2142.125613] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2142.127165] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2142.130202] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2142.132587] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2142, 0, 2192 [ 2142.138804] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2142.140874] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2142.143346] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2142.148462] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2142, 0, 2192 [ 2142.153709] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2149.678258] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 23:42:34 (1788666154) [ 2150.188277] LNet: 107863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.195646] LNetError: 107863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2150.206333] LNet: Removed LNI 192.168.204.21@tcp [ 2151.048414] Key type .llcrypt unregistered [ 2151.050061] Key type ._llcrypt unregistered [ 2157.469579] Key type ._llcrypt registered [ 2157.477320] Key type .llcrypt registered [ 2157.721358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2157.742601] alg: No test for adler32 (adler32-zlib) [ 2158.777175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2158.838124] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2168.324748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.121@tcp [ 2179.519621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2179.575573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2179.601552] LNet: Added LNI 192.168.204.21@tcp1 [8/256/0/180] [ 2180.621216] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2180.820031] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2180.825223] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2180.829609] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2183.122268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2183.162931] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2196.634702] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 23:43:20 (1788666200) [ 2196.956742] LNet: 109365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.996788] LNetError: 109365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2197.010880] LNet: Removed LNI 192.168.204.21@tcp [ 2198.051224] LNet: Removed LNI 192.168.204.21@tcp1 [ 2198.053182] LNetError: 109365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2198.068047] LNetError: 109365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2198.091822] LNet: Removed LNI 10.1.2.3@tcp99 [ 2198.608653] Key type .llcrypt unregistered [ 2198.614340] Key type ._llcrypt unregistered [ 2206.185929] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2206.190172] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2208.748634] Key type ._llcrypt registered [ 2208.751419] Key type .llcrypt registered [ 2208.989404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2209.025229] alg: No test for adler32 (adler32-zlib) [ 2209.994917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.064143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2210.096983] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2210.960766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2210.995322] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2213.637812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2213.710967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2213.765368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2213.787158] LNetError: 110107:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2213.821068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2213.883131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2214.421380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2214.535708] LNet: 110128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.544830] LNetError: 110128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2214.554168] LNet: Removed LNI 10.1.2.3@tcp [ 2215.594801] LNet: Removed LNI 192.168.204.21@tcp [ 2216.349574] Key type .llcrypt unregistered [ 2216.351169] Key type ._llcrypt unregistered [ 2230.371197] Key type ._llcrypt registered [ 2230.375782] Key type .llcrypt registered [ 2231.553884] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 23:43:56 (1788666236) [ 2232.403917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2232.420764] alg: No test for adler32 (adler32-zlib) [ 2233.341034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2233.387288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2233.408502] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2234.260076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2234.335972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg421-client@tcp [ 2241.647602] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 23:44:06 (1788666246) [ 2241.799758] LNet: 111651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2241.808750] LNetError: 111651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2241.829827] LNet: Removed LNI 192.168.204.21@tcp [ 2242.833493] Key type .llcrypt unregistered [ 2242.838873] Key type ._llcrypt unregistered [ 2250.272050] Key type ._llcrypt registered [ 2250.275528] Key type .llcrypt registered [ 2250.497607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2250.525990] alg: No test for adler32 (adler32-zlib) [ 2251.483277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2251.512394] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2251.535909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2258.377539] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 23:44:23 (1788666263) [ 2260.178990] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2262.085523] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 23:44:26 (1788666266) [ 2263.698728] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2265.861429] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 23:44:30 (1788666270) [ 2266.193793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2266.223279] LNet: 113081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2266.249500] LNetError: 113081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2266.283516] LNet: Removed LNI 192.168.204.21@tcp [ 2266.347531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2266.415444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2266.445913] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2267.144526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2267.208972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2276.414784] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 23:44:40 (1788666280) [ 2276.567845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2276.639668] LNet: 113785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.658416] LNetError: 113785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.690549] LNet: Removed LNI 192.168.204.21@tcp [ 2276.788798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2276.908847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2276.978576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2277.715419] Key type .llcrypt unregistered [ 2277.719067] Key type ._llcrypt unregistered [ 2290.297566] Key type ._llcrypt registered [ 2290.301685] Key type .llcrypt registered [ 2291.756995] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 23:44:56 (1788666296) [ 2292.592464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2292.611825] alg: No test for adler32 (adler32-zlib) [ 2293.574421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.645450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2293.679837] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2294.566957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2294.677237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2302.717930] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 23:45:07 (1788666307) [ 2302.843635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2302.884159] LNet: 115342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.891938] LNetError: 115342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.933034] LNet: Removed LNI 192.168.204.21@tcp [ 2302.999207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2303.062961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2303.095653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2310.745701] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 23:45:15 (1788666315) [ 2311.016860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.034624] LNet: 115925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.070402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.148316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2311.622678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2311.657616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2311.702748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2312.150737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2312.183577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2312.257314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2312.284947] LNet: 115958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.321050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.405930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2312.886152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2312.960182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2313.071748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2313.459993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2313.490283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2320.746104] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 23:45:25 (1788666325) [ 2321.203562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2321.220211] LNet: 116558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2321.247651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2321.334521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2321.362100] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2322.157385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2322.181555] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 2323.124965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2323.222574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2323.552810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2323.582569] LNetError: 116838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2323.586380] LNet: Removed LNI 192.168.204.21@tcp [ 2323.594964] LNet: Removed LNI 192.168.204.21@tcp2 [ 2323.625380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.717556] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2323.722852] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 2323.728592] LNet: 116844:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2324.377497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2324.696064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2324.741797] LNetError: 116881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.751195] LNetError: 116881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2324.770762] LNet: Removed LNI 192.168.204.21@tcp [ 2324.793037] LNet: Removed LNI 192.168.204.21@tcp2 [ 2324.820077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2324.881505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2324.907523] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2325.860966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2325.902700] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 2326.891474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2327.023721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2327.154922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2327.367582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2327.404479] LNet: 117166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2327.414420] LNet: 117166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2327.420880] LNetError: 117166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2327.429113] LNetError: 117166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2327.443443] LNet: Removed LNI 192.168.204.21@tcp [ 2328.487321] LNet: Removed LNI 192.168.204.21@tcp2 [ 2328.561397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.688781] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2328.695847] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 2328.701283] LNet: 117172:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2329.411724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2329.654478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2329.677873] LNet: Removed LNI 192.168.204.21@tcp [ 2330.730273] LNet: Removed LNI 192.168.204.21@tcp2 [ 2330.776295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2330.839634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2330.869695] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2331.657071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2331.674575] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 2332.574476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2332.638265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2332.891505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2332.936132] LNetError: 117489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2332.941788] LNetError: 117489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2332.950128] LNet: Removed LNI 192.168.204.21@tcp [ 2332.956743] LNet: Removed LNI 192.168.204.21@tcp2 [ 2332.997802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.073427] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2333.078915] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 2333.083909] LNet: 117495:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2333.728972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2333.980532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.007498] LNet: Removed LNI 192.168.204.21@tcp [ 2335.032516] LNet: Removed LNI 192.168.204.21@tcp2 [ 2335.068433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.166514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2335.190660] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2335.976597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2335.997717] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 2336.946586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2336.986387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2337.054379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2337.366947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2337.390701] LNet: 117816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.394781] LNet: 117816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2338.463378] LNet: Removed LNI 192.168.204.21@tcp [ 2339.498234] LNet: Removed LNI 192.168.204.21@tcp2 [ 2339.551374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2339.646501] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2339.656593] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 2339.661647] LNet: 117823:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2340.098538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2340.366879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.400981] LNet: Removed LNI 192.168.204.21@tcp [ 2341.419868] LNet: Removed LNI 192.168.204.21@tcp2 [ 2341.462752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2341.561641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2341.594400] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2342.257459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2342.276422] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 2343.035994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2343.073279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2343.123803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2343.445312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.466897] LNetError: 118157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2343.475409] LNetError: 118157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2343.481854] LNet: Removed LNI 192.168.204.21@tcp [ 2343.488416] LNet: Removed LNI 192.168.204.21@tcp2 [ 2343.520188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.560891] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2343.568973] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 2344.017492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2344.368355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2345.439214] LNet: Removed LNI 192.168.204.21@tcp [ 2345.456238] LNet: Removed LNI 192.168.204.21@tcp2 [ 2345.482748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2345.574463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2345.615514] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2346.493635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2346.518024] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 2347.616064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2347.655210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2347.685194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2347.728763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2348.131072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2349.152137] LNet: Removed LNI 192.168.204.21@tcp [ 2350.188435] LNet: Removed LNI 192.168.204.21@tcp2 [ 2350.283283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.338368] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2350.345565] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 2350.353474] LNet: 118523:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2350.357966] LNet: 118523:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 2350.956592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2360.780984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2363.630584] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 23:46:07 (1788666367) [ 2363.881439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2363.944130] LNet: 119276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2363.957163] LNet: 119276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2363.972299] LNetError: 119276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2363.986813] LNetError: 119276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2364.012777] LNet: Removed LNI 192.168.204.21@tcp [ 2365.043629] LNet: Removed LNI 192.168.204.21@tcp2 [ 2365.207372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2366.146968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2366.199051] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2367.134817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2367.183529] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2368.793413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2368.816506] LNet: Removed LNI 10.1.2.3@tcp [ 2368.840582] LNet: Removed LNI 192.168.204.21@tcp [ 2368.868626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.032486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2369.092854] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2369.114818] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2370.329775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2370.400596] LNet: Removed LNI 192.168.204.21@tcp [ 2370.413731] LNet: Removed LNI 10.1.2.3@tcp [ 2379.712331] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 23:46:24 (1788666384) [ 2380.628241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2380.733276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.869237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2380.928361] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2382.324088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2382.379428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2382.434631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2382.489758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2382.541368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2382.624771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2382.675698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2382.708330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2382.771082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2382.865876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2382.943828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2382.996314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2383.283690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2389.791496] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 23:46:34 (1788666394) [ 2390.134767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2391.207183] LNet: Removed LNI 192.168.204.21@tcp [ 2391.264934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2391.322895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2391.363379] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2392.038087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2392.073792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2392.122529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2392.159164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2392.197763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2392.238862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2392.277417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2392.316452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2392.365951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2392.425439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2392.478980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2392.534088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2392.637028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2392.697933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2392.756178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2392.790942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2392.842166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2392.913371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2392.975757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2393.024788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2393.098231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2393.173943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2393.266798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.21@tcp [ 2400.441288] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 23:46:45 (1788666405) [ 2400.641983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2400.671487] LNet: 121650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.685283] LNet: 121650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2400.710214] LNetError: 121650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.715585] LNetError: 121650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2400.762749] LNet: Removed LNI 192.168.204.21@tcp [ 2400.793679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2400.942926] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2401.180138] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2401.183623] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2401.188262] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2403.524779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2403.562550] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2404.220291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2404.234053] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2405.943362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2406.017421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2406.097753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.21@tcp -m PUT -e local_timeout -r 1 [ 2406.169393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2406.232752] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2406.235872] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2406.536398] LNetError: 121721:0:(lib-move.c:4794:lnet_parse()) 192.168.204.21@tcp, src 192.168.204.21@tcp: Dropping PUT (error -108 looking up sender) [ 2407.585566] LNet: Removed LNI 10.1.2.3@tcp [ 2408.615737] LNet: Removed LNI 192.168.204.21@tcp [ 2409.091904] Key type .llcrypt unregistered [ 2409.101308] Key type ._llcrypt unregistered [ 2423.801154] Key type ._llcrypt registered [ 2423.805843] Key type .llcrypt registered [ 2425.504705] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 23:47:09 (1788666429) [ 2425.874387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2425.884789] alg: No test for adler32 (adler32-zlib) [ 2426.865946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2427.295460] LNet: 123030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2428.101689] Key type .llcrypt unregistered [ 2428.104078] Key type ._llcrypt unregistered [ 2435.057952] Key type ._llcrypt registered [ 2435.060450] Key type .llcrypt registered [ 2435.300733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2435.321421] alg: No test for adler32 (adler32-zlib) [ 2436.279907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2436.312781] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2445.403770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.121@tcp [ 2445.636564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2446.178241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.121@tcp [ 2446.415506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2446.443418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2446.467903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2446.489725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2446.518700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2446.551148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2446.559196] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.121@tcp [ 2446.607299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2446.644930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2446.702173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2446.730145] LNet: 123479:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2446.763102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2446.792203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2446.835345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2446.876430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2446.947925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2446.992189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2447.677582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2447.990792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2448.030041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2448.084148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2448.119960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2448.154340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2448.207537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2448.248395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2448.271597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2448.303823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2448.344102] LNet: 123598:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2448.386445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2448.424703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2448.472014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2448.501968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2448.541913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2448.584730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2449.176373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.121@tcp [ 2449.395857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2449.419831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2449.446984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2449.473133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2449.502554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2449.547334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2449.615384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2449.674972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2449.718439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2449.732958] LNet: 123718:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2449.750153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2449.782157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2449.808422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2449.832703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2449.855571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2449.880861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2450.522443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2450.684671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2450.704607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2450.738251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2450.776423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2450.811637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2450.849488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2450.893972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2450.964591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2451.003763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2451.056609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2451.115939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2451.151139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2451.211936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2451.258472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2451.310150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2451.701707] LNet: 123883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2451.712113] LNetError: 123883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2451.737449] LNet: Removed LNI 192.168.204.21@tcp [ 2452.327731] Key type .llcrypt unregistered [ 2452.334802] Key type ._llcrypt unregistered [ 2472.795427] Key type ._llcrypt registered [ 2472.798581] Key type .llcrypt registered [ 2474.340788] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 23:47:58 (1788666478) [ 2475.314336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2475.339521] alg: No test for adler32 (adler32-zlib) [ 2476.347187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2477.096087] LNet: 124769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2477.847712] Key type .llcrypt unregistered [ 2477.852135] Key type ._llcrypt unregistered [ 2486.304186] Key type ._llcrypt registered [ 2486.305954] Key type .llcrypt registered [ 2486.538655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2486.558459] alg: No test for adler32 (adler32-zlib) [ 2487.488075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2487.511982] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2497.154495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.121@tcp [ 2506.604604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2506.662306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2506.693833] LNet: Added LNI 192.168.204.21@tcp1 [8/256/0/180] [ 2507.682594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2508.273220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.121@tcp [ 2508.542219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2508.575355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2508.662970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2508.723305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2508.771246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2508.837866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2508.922919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2508.961253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2508.997529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2509.017960] LNet: 125420:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2509.026802] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.121@tcp [ 2509.047175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2509.082495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2509.133763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2509.174790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2509.206721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2509.241525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2510.012277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2510.217844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2510.250441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2510.278412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2510.304844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2510.336935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2510.371848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2510.409434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2510.441935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2510.468337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2510.485944] LNet: 125539:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2510.506687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2510.533964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2510.560753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2510.600789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2510.646570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2510.710885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2511.481698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.121@tcp [ 2511.761205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2511.796406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2511.835398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2511.871391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2511.917135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2511.969357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2512.011956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2512.061744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2512.123483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2512.143622] LNet: 125659:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2512.162837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2512.196071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2512.257930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2512.294914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2512.359714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2512.434908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2513.250929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2513.550738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2513.631629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2513.679553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2513.716868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2513.742398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2513.785595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2513.816560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2513.848409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2513.884952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2513.932749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2513.988432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2514.021643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2514.059911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2514.099624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2514.138972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2519.416862] LNet: 125859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2519.428917] LNetError: 125859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2519.439570] LNet: Removed LNI 192.168.204.21@tcp [ 2519.468542] LNet: Removed LNI 192.168.204.21@tcp1 [ 2519.885723] Key type .llcrypt unregistered [ 2519.901280] Key type ._llcrypt unregistered [ 2537.339616] Key type ._llcrypt registered [ 2537.343622] Key type .llcrypt registered [ 2538.506989] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 23:49:03 (1788666543) [ 2538.900442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2538.919574] alg: No test for adler32 (adler32-zlib) [ 2539.876570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2539.921182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2539.943323] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2540.808876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2540.826930] LNet: Added LNI 192.168.204.21@tcp2 [8/256/0/180] [ 2541.743653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.21@tcp [ 2541.884536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2542.944823] LNet: Removed LNI 192.168.204.21@tcp [ 2549.979586] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 23:49:14 (1788666554) [ 2551.596727] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2553.085982] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 23:49:18 (1788666558) [ 2559.855613] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 23:49:24 (1788666564) [ 2567.213972] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 23:49:31 (1788666571) [ 2574.436569] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 23:49:39 (1788666579) [ 2582.134462] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 23:49:46 (1788666586) [ 2589.583365] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 23:49:54 (1788666594) [ 2597.143239] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 23:50:02 (1788666602) [ 2605.188164] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 23:50:09 (1788666609) [ 2612.930895] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 23:50:17 (1788666617) [ 2620.364413] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 23:50:25 (1788666625) [ 2627.096618] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 23:50:32 (1788666632) [ 2634.501855] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 23:50:39 (1788666639) [ 2642.301685] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 23:50:47 (1788666647) [ 2649.511967] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 23:50:54 (1788666654) [ 2657.088666] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 23:51:02 (1788666662) [ 2664.450928] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 23:51:09 (1788666669) [ 2671.974536] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 23:51:16 (1788666676) [ 2678.185491] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 23:51:23 (1788666683) [ 2678.421618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2678.442791] LNet: 137551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2678.453720] LNetError: 137551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2679.526245] LNet: Removed LNI 192.168.204.21@tcp2 [ 2679.555379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2679.661104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2679.682381] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2680.488315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2680.519142] LNet: 137693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2680.522308] LNetError: 137693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2680.533712] LNet: Removed LNI 192.168.204.21@tcp [ 2680.546034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2680.572214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2680.587876] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2686.664247] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 23:51:31 (1788666691) [ 2693.675298] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 23:51:38 (1788666698) [ 2701.944992] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 23:51:46 (1788666706) [ 2710.529553] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 23:51:54 (1788666714) [ 2718.869718] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 23:52:03 (1788666723) [ 2726.433706] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 23:52:11 (1788666731) [ 2732.516566] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 23:52:17 (1788666737) [ 2738.871836] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 23:52:23 (1788666743) [ 2744.724700] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 23:52:29 (1788666749) [ 2750.856582] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 23:52:36 (1788666756) [ 2759.225367] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 23:52:43 (1788666763) [ 2766.921125] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 23:52:51 (1788666771) [ 2774.382376] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2524 sec ========== 23:52:59 (1788666779) [ 2774.790865] LNet: 145160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2774.805597] LNetError: 145160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2774.821241] LNet: Removed LNI 192.168.204.21@tcp [ 2775.549775] Key type .llcrypt unregistered [ 2775.554048] Key type ._llcrypt unregistered [ 2789.163571] Key type ._llcrypt registered [ 2789.193755] Key type .llcrypt registered [ 2789.746040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2789.761996] alg: No test for adler32 (adler32-zlib) [ 2790.920677] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 2791.209858] LNet: Added LNI 192.168.204.21@tcp [8/256/0/180] [ 2792.911239] Key type lgssc registered [ 2794.194474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2863.817554] Lustre: Mounted lustre-client - version 2.17.58_4_g3b30273 [ 2869.564582] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2879.047729] Lustre: Unmounted lustre-client [ 2905.023420] Key type lgssc unregistered [ 2905.337481] LNet: 148111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2905.344685] LNetError: 148111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2905.369907] LNet: Removed LNI 192.168.204.21@tcp [ 2905.999157] Key type .llcrypt unregistered [ 2906.000735] Key type ._llcrypt unregistered