[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 645803204 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003216] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.006046] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011158] LSM: Security Framework initializing [ 0.012243] Yama: becoming mindful. [ 0.013059] SELinux: Initializing. [ 0.014115] *** VALIDATE selinux *** [ 0.020666] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025504] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027069] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028131] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029129] *** VALIDATE tmpfs *** [ 0.031370] *** VALIDATE proc *** [ 0.033389] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035281] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038028] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.041497] debug: unmapping init [mem 0xffffffff83e59000-0xffffffff83e60fff] [ 0.043103] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044634] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045019] ... version: 2 [ 0.046007] ... bit width: 48 [ 0.047009] ... generic registers: 4 [ 0.048008] ... value mask: 0000ffffffffffff [ 0.049008] ... max period: 00007fffffffffff [ 0.050011] ... fixed-purpose events: 3 [ 0.051008] ... event mask: 000000070000000f [ 0.053095] rcu: Hierarchical SRCU implementation. [ 0.055422] smp: Bringing up secondary CPUs ... [ 0.056537] x86: Booting SMP configuration: [ 0.057017] .... node #0, CPUs: #1 #2 #3 [ 0.064080] smp: Brought up 1 node, 4 CPUs [ 0.066011] smpboot: Max logical packages: 1 [ 0.067010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.250650] node 0 deferred pages initialised in 181ms [ 0.254134] devtmpfs: initialized [ 0.255000] x86/mm: Memory block size: 128MB [ 0.257735] gcov: version magic: 0x41383552 [ 0.259242] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.262098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.264362] pinctrl core: initialized pinctrl subsystem [ 0.265167] [ 0.265753] ************************************************************* [ 0.268015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.270012] ** ** [ 0.272013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.274011] ** ** [ 0.276014] ** This means that this kernel is built to expose internal ** [ 0.278013] ** IOMMU data structures, which may compromise security on ** [ 0.281031] ** your system. ** [ 0.284022] ** ** [ 0.287017] ** If you see this message and you are not debugging the ** [ 0.289017] ** kernel, report this immediately to your vendor! ** [ 0.291021] ** ** [ 0.292010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.294012] ************************************************************* [ 0.296729] NET: Registered protocol family 16 [ 0.298462] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.301071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.303069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.307052] cpuidle: using governor menu [ 0.308867] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.309704] PCI: Using configuration type 1 for base access [ 0.310140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.320096] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.322017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.325248] cryptd: max_cpu_qlen set to 1000 [ 0.328492] ACPI: Added _OSI(Module Device) [ 0.330018] ACPI: Added _OSI(Processor Device) [ 0.332013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.333014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.337387] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.347322] ACPI: Interpreter enabled [ 0.349056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.350015] ACPI: Using IOAPIC for interrupt routing [ 0.351094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.353395] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.363698] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.366041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.369020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.372092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.376261] acpiphp: Slot [2] registered [ 0.378117] acpiphp: Slot [3] registered [ 0.379100] acpiphp: Slot [4] registered [ 0.381096] acpiphp: Slot [5] registered [ 0.382089] acpiphp: Slot [6] registered [ 0.384087] acpiphp: Slot [7] registered [ 0.385079] acpiphp: Slot [8] registered [ 0.386076] acpiphp: Slot [9] registered [ 0.388109] acpiphp: Slot [10] registered [ 0.389092] acpiphp: Slot [11] registered [ 0.392117] acpiphp: Slot [12] registered [ 0.393078] acpiphp: Slot [13] registered [ 0.395096] acpiphp: Slot [14] registered [ 0.396106] acpiphp: Slot [15] registered [ 0.398089] acpiphp: Slot [16] registered [ 0.399091] acpiphp: Slot [17] registered [ 0.401086] acpiphp: Slot [18] registered [ 0.402090] acpiphp: Slot [19] registered [ 0.403070] acpiphp: Slot [20] registered [ 0.405078] acpiphp: Slot [21] registered [ 0.406068] acpiphp: Slot [22] registered [ 0.407092] acpiphp: Slot [23] registered [ 0.409089] acpiphp: Slot [24] registered [ 0.410097] acpiphp: Slot [25] registered [ 0.412107] acpiphp: Slot [26] registered [ 0.413081] acpiphp: Slot [27] registered [ 0.415091] acpiphp: Slot [28] registered [ 0.416083] acpiphp: Slot [29] registered [ 0.417091] acpiphp: Slot [30] registered [ 0.419111] acpiphp: Slot [31] registered [ 0.421069] PCI host bridge to bus 0000:00 [ 0.422014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.425019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.426021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.429054] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.431025] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.434024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.435133] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.439313] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.440529] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.449015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.453014] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.456016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.458016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.460027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.463600] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.465860] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.468042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.470639] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.474012] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.481000] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.485015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.489000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.495018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.499017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.513019] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.520633] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.532023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.537017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.556016] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.566693] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.571666] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.573522] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.575650] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.578298] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.584100] iommu: Default domain type: Passthrough [ 0.585403] SCSI subsystem initialized [ 0.587219] ACPI: bus type USB registered [ 0.589114] usbcore: registered new interface driver usbfs [ 0.591102] usbcore: registered new interface driver hub [ 0.593078] usbcore: registered new device driver usb [ 0.594170] pps_core: LinuxPPS API ver. 1 registered [ 0.596023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.599064] PTP clock support registered [ 0.601190] EDAC MC: Ver: 3.0.0 [ 0.603118] PCI: Using ACPI for IRQ routing [ 0.605851] NetLabel: Initializing [ 0.607015] NetLabel: domain hash size = 128 [ 0.608010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.610090] NetLabel: unlabeled traffic allowed by default [ 0.612103] vgaarb: loaded [ 0.613258] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.615013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.621063] clocksource: Switched to clocksource kvm-clock [ 0.728753] VFS: Disk quotas dquot_6.6.0 [ 0.730140] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.732637] *** VALIDATE ramfs *** [ 0.733887] *** VALIDATE hugetlbfs *** [ 0.735562] pnp: PnP ACPI init [ 0.737855] pnp: PnP ACPI: found 6 devices [ 0.757496] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.761295] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.763579] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.765867] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.768599] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.770913] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.773495] NET: Registered protocol family 2 [ 0.775584] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.780242] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.783709] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.788945] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.792032] TCP: Hash tables configured (established 65536 bind 65536) [ 0.793879] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.797032] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.800232] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.803168] NET: Registered protocol family 1 [ 0.805946] RPC: Registered named UNIX socket transport module. [ 0.808649] RPC: Registered udp transport module. [ 0.810594] RPC: Registered tcp transport module. [ 0.812529] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.815410] NET: Registered protocol family 44 [ 0.817088] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.819490] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.821810] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.824353] PCI: CLS 0 bytes, default 64 [ 0.826241] Unpacking initramfs... [ 2.223691] debug: unmapping init [mem 0xffff88a23cc64000-0xffff88a23ffcffff] [ 2.227672] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.229600] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.232270] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.778661] Initialise system trusted keyrings [ 2.780296] Key type blacklist registered [ 2.781806] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.796197] zbud: loaded [ 2.802794] *** VALIDATE nfs *** [ 2.804520] *** VALIDATE nfs4 *** [ 2.806464] pstore: using deflate compression [ 2.809955] Platform Keyring initialized [ 2.937320] NET: Registered protocol family 38 [ 2.938737] Key type asymmetric registered [ 2.940218] Asymmetric key parser 'x509' registered [ 2.941684] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.945740] io scheduler mq-deadline registered [ 2.947568] io scheduler kyber registered [ 2.949639] io scheduler bfq registered [ 2.953420] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.956101] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.963295] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.965947] ACPI: Power Button [PWRF] [ 3.075330] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.167223] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.271710] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.301948] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.344847] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.356623] Non-volatile memory driver v1.3 [ 3.357871] Linux agpgart interface v0.103 [ 3.421925] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.428575] vda: detected capacity change from 0 to 68526080 [ 3.461477] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.465111] vdb: detected capacity change from 0 to 1073741824 [ 3.479136] libphy: Fixed MDIO Bus: probed [ 3.493211] usbcore: registered new interface driver usbserial_generic [ 3.495708] usbserial: USB Serial support registered for generic [ 3.503408] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.512190] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.514132] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.524546] mousedev: PS/2 mouse device common for all mice [ 3.528818] rtc_cmos 00:05: RTC can wake from S4 [ 3.536429] rtc_cmos 00:05: registered as rtc0 [ 3.538729] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.543517] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.543560] intel_pstate: CPU model not supported [ 3.548683] hid: raw HID events driver (C) Jiri Kosina [ 3.560613] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.564870] usbcore: registered new interface driver usbhid [ 3.564877] usbhid: USB HID core driver [ 3.564997] drop_monitor: Initializing network drop monitor service [ 3.565137] Initializing XFRM netlink socket [ 3.565548] NET: Registered protocol family 10 [ 3.566771] Segment Routing with IPv6 [ 3.566805] NET: Registered protocol family 17 [ 3.567099] mpls_gso: MPLS GSO support [ 3.588763] RAS: Correctable Errors collector initialized. [ 3.590658] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.593275] AVX version of gcm_enc/dec engaged. [ 3.593283] AES CTR mode by8 optimization enabled [ 3.728557] sched_clock: Marking stable (3728537017, 0)->(4624158360, -895621343) [ 3.732895] registered taskstats version 1 [ 3.734514] Loading compiled-in X.509 certificates [ 3.738367] zswap: loaded using pool lzo/zbud [ 3.778943] Key type big_key registered [ 3.802785] Key type encrypted registered [ 3.804277] ima: No TPM chip found, activating TPM-bypass! [ 3.806370] ima: Allocated hash algorithm: sha1 [ 3.808103] ima: No architecture policies found [ 3.809661] evm: Initialising EVM extended attributes: [ 3.811167] evm: security.selinux [ 3.812442] evm: security.ima [ 3.813589] evm: security.capability [ 3.814886] evm: HMAC attrs: 0x1 [ 3.818755] rtc_cmos 00:05: setting system clock to 2025-11-16 22:24:52 UTC (1763331892) [ 3.831413] debug: unmapping init [mem 0xffffffff84e03000-0xffffffff84ffffff] [ 3.833904] debug: unmapping init [mem 0xffffffff83b82000-0xffffffff83e58fff] [ 3.842598] Write protecting the kernel read-only data: 28672k [ 3.853615] debug: unmapping init [mem 0xffffffff82203000-0xffffffff823fffff] [ 3.856364] debug: unmapping init [mem 0xffffffff82b14000-0xffffffff82bfffff] [ 3.895232] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.904891] systemd[1]: Detected virtualization kvm. [ 3.907202] systemd[1]: Detected architecture x86-64. [ 3.909662] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.944472] systemd[1]: No hostname configured. [ 3.946598] systemd[1]: Set hostname to . [ 3.949023] random: systemd: uninitialized urandom read (16 bytes read) [ 3.951695] systemd[1]: Initializing machine ID from random generator. [ 4.057447] random: ln: uninitialized urandom read (6 bytes read) [ 4.347303] random: systemd: uninitialized urandom read (16 bytes read) [ 4.361297] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.378410] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.392337] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.697480] device-mapper: uevent: version 1.0.3 [ 5.699863] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.215531] virtio_net virtio0 ens2: renamed from eth0 [ 7.831216] scsi host0: ata_piix [ 7.918187] scsi host1: ata_piix [ 7.919828] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.922039] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.351050] random: crng init done [ 12.359186] random: 7 urandom warning(s) missed due to ratelimiting [ 15.547032] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.822924] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ 18.200562] hrtimer: interrupt took 3594133 ns [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.871990] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.348203] SELinux: Disabled at runtime. [ 19.418957] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.437230] systemd[1]: Detected virtualization kvm. [ 19.439206] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.313723] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.319432] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.326502] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.330341] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.333840] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.348347] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.366128] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ 20.868373] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 21.730645] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.653086] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.660994] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.500513] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.559267] EDAC sbridge: Ver: 1.1.2 [ 25.710111] Key type dns_resolver registered [ 26.182138] NFS: Registering the id_resolver key type [ 26.183710] Key type id_resolver registered [ 26.185116] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg349-client login: [ 87.488927] libcfs: loading out-of-tree module taints kernel. [ 87.517322] Key type ._llcrypt registered [ 87.541955] Key type .llcrypt registered [ 87.893624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 87.902480] alg: No test for adler32 (adler32-zlib) [ 89.212942] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 89.818630] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 91.487160] Key type lgssc registered [ 92.530468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 199.678566] Lustre: Mounted lustre-client [ 204.151858] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 220.655154] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing check_logdir /tmp/testlogs/ [ 224.512681] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing yml_node [ 225.251057] Lustre: lustre-OST0000-osc-ffff88a285da7800: disconnect after 23s idle [ 228.483441] Lustre: DEBUG MARKER: Client: 2.16.61.53 [ 230.914734] Lustre: DEBUG MARKER: MDS: 2.16.61.53 [ 233.403526] Lustre: DEBUG MARKER: OSS: 2.16.61.53 [ 235.117859] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Nov 16 17:28:42 EST 2025 [ 248.607081] LustreError: 7911:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff88a285da7800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 248.616236] LustreError: 7911:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 248.649099] Lustre: Unmounted lustre-client [ 274.936397] Key type lgssc unregistered [ 275.230951] LNet: 8357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 275.246640] LNetError: 8357:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 275.263346] LNet: Removed LNI 192.168.203.49@tcp [ 275.902132] Key type .llcrypt unregistered [ 275.903858] Key type ._llcrypt unregistered [ 297.585765] Key type ._llcrypt registered [ 297.588097] Key type .llcrypt registered [ 297.854489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 297.872772] alg: No test for adler32 (adler32-zlib) [ 298.812567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 298.878727] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 298.912387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 302.658759] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 302.799195] LNet: 9226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.815753] LNetError: 9226:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 302.853124] LNet: Removed LNI 192.168.203.49@tcp [ 303.431366] Key type .llcrypt unregistered [ 303.441387] Key type ._llcrypt unregistered [ 307.824120] Key type ._llcrypt registered [ 307.825763] Key type .llcrypt registered [ 308.993205] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 17:29:56 (1763332196) [ 309.299784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 309.310143] alg: No test for adler32 (adler32-zlib) [ 310.214813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.297822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.330796] LNet: 9636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.368468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.425339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 310.446784] LNet: 9643:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 316.958956] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 17:30:04 (1763332204) [ 317.277710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.305386] LNet: 10220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.327134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.388400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 324.096839] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 17:30:11 (1763332211) [ 324.180175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.225440] LNet: 10798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.266660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.411841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 330.618416] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 17:30:17 (1763332217) [ 330.764698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.793757] LNet: 11387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.826308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.897276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 337.016814] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 17:30:24 (1763332224) [ 337.184834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.226560] LNet: 11974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.270883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.359708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 337.789801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 344.701985] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 17:30:31 (1763332231) [ 344.780802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.844970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.913940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 345.270650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 345.761707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 352.118270] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 17:30:39 (1763332239) [ 352.415485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.429189] LNet: 13164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.432220] LNet: 13164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 352.456293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.515264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 359.341313] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 17:30:46 (1763332246) [ 359.430926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.496171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.581281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 359.618237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 360.404455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 360.455120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 360.900391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 360.944655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 361.296891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 361.319484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 361.997128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 362.055768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 362.787318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 362.864942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 363.638650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 363.705399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 364.521166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 364.583260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 365.172422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 365.225319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 365.840817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 365.928534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 366.189898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 366.211532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 366.632971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 366.705170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 367.155951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 367.205228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 373.988592] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 17:31:01 (1763332261) [ 374.312577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.333664] LNet: 14430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.339763] LNet: 14430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 374.391189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.534855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 374.559452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 381.250329] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 17:31:08 (1763332268) [ 381.478722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.562840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.709667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 381.759814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 388.617247] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 17:31:16 (1763332276) [ 388.778787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.816397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.892343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 388.930173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 394.720761] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 17:31:21 (1763332281) [ 394.810684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.872577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.958401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 394.985971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 400.673280] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 17:31:28 (1763332288) [ 400.806392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.864103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.947062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 400.981251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 407.089126] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 17:31:34 (1763332294) [ 407.305311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.327014] LNet: 17362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.331272] LNet: 17362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 407.345100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.416719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 407.455390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 412.795946] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 17:31:40 (1763332300) [ 412.996434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.023888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.123493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 413.570850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 418.817565] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 17:31:46 (1763332306) [ 419.008378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.063975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.220908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 419.427457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 425.821279] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 17:31:53 (1763332313) [ 425.877792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.943688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.113369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 426.401709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 432.657737] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 17:31:59 (1763332319) [ 432.792181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.845474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.251362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 433.482544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 439.263610] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 17:32:06 (1763332326) [ 439.518435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.564273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.641854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 439.703162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 445.871772] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 17:32:13 (1763332333) [ 445.955178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.983318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.065774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 446.133530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 453.316244] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 17:32:20 (1763332340) [ 453.364751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.416888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.511332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 460.066459] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 17:32:27 (1763332347) [ 460.378739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.448664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.577431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 467.051487] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 17:32:34 (1763332354) [ 467.365895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.398443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.483028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 467.923457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 474.537693] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 17:32:42 (1763332362) [ 474.645518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.660873] LNet: 23382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.663915] LNet: 23382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 474.680307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.771583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 474.795096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 482.561856] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 17:32:49 (1763332369) [ 482.990232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.053509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.206158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 483.260834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 491.224799] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 17:32:58 (1763332378) [ 491.362422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.450520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.606302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 491.655437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 500.579965] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 17:33:07 (1763332387) [ 500.692825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.758103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.852509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 500.916384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 507.575426] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 17:33:14 (1763332394) [ 507.844423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.892430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.928107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 507.998435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 508.035566] LNetError: 25745:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 508.078388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 508.114505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 514.678878] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 17:33:22 (1763332402) [ 515.044989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.107494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.229524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.oLDMm29Ag8 [ 515.308252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.oLDMm29Ag8 [ 515.346456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.oLDMm29Ag8 [ 521.854325] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 17:33:28 (1763332408) [ 521.950851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.995423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.101503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 522.135706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 522.184871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 527.879416] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 17:33:35 (1763332415) [ 528.083599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.138754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.236282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 531.206452] LNet: Message forwarding will be enabled in 2s [ 534.305128] LNet: Message forwarding enabled [ 534.726770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 534.752848] LNet: Message forwarding will stop in 2s to 6s [ 539.382477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 540.032831] LNet: Message forwarding disabled [ 542.171238] LNet: Message forwarding will be enabled in 2s [ 546.211172] LNet: Message forwarding enabled [ 549.707140] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 17:33:56 (1763332436) [ 549.826574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.119208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.204397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 556.997637] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 17:34:04 (1763332444) [ 557.090487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.160325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.269686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 557.305220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 557.336764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 557.370950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 557.416096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 557.442536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 557.466564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 557.494465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 557.532831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 557.558382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 557.584764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 557.602842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 557.636625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 557.671145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 557.716942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 557.756498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 557.806356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 557.844289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 557.885263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 557.919132] LNetError: 28814:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 563.780943] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 17:34:11 (1763332451) [ 564.038463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.065303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.139275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 570.549996] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 17:34:17 (1763332457) [ 572.256843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg349-server [ 574.023809] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 17:34:21 (1763332461) [ 574.119668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.199186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.260314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 574.289914] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 576.218543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[50-54]@tcp [ 576.258449] LNet: 30530:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 576.326601] LNetError: 30318:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.50@tcp is being used as a gateway but routing feature is not turned on [ 576.356919] LNetError: 30318:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.50@tcp has gone from up to down [ 577.314651] LNetError: 30318:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.50@tcp is being used as a gateway but routing feature is not turned on [ 577.326440] LNetError: 30318:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 578.351457] LNetError: 30318:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.51@tcp is being used as a gateway but routing feature is not turned on [ 578.360915] LNetError: 30318:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 580.410301] LNetError: 30318:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.51@tcp is being used as a gateway but routing feature is not turned on [ 580.421262] LNetError: 30318:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 582.551854] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 17:34:29 (1763332469) [ 584.145592] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg349-server [ 584.502450] LNetError: 30318:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.51@tcp is being used as a gateway but routing feature is not turned on [ 584.518892] LNetError: 30318:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 585.643956] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 17:34:33 (1763332473) [ 585.837475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.895287] LNetError: 31464:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 586.925848] LNet: Removed LNI 192.168.203.49@tcp [ 586.945658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.009284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 587.030385] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 587.666640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[50-55/2]@tcp [ 587.699784] LNet: 31582:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 587.709033] LNet: 31582:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 587.728322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[50-55/2]@tcp [ 587.729284] LNetError: 31468:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.50@tcp has gone from up to down [ 587.765196] LNetError: 31468:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 593.401619] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 17:34:40 (1763332480) [ 593.579451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.616902] LNetError: 32174:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.635353] LNet: Removed LNI 192.168.203.49@tcp [ 593.660259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.741824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 593.783606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.819508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.864564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 593.909871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.936429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.991610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 594.058848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.096255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.152230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 594.207963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.231767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.270907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 594.340493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.369937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.416862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 594.499068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.529522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.565975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 594.635745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.672708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.751209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 594.854329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.894903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.952987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 595.014424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.059881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.146337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 595.220315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.262483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.319712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 595.372843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.404851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.460726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 601.462930] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 17:34:48 (1763332488) [ 603.237453] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg349-server [ 604.954522] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 17:34:52 (1763332492) [ 606.651989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg349-server [ 608.298761] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 17:34:55 (1763332495) [ 608.577441] LNet: 33575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.584674] LNet: 33575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 24 previous similar messages [ 609.221858] Key type .llcrypt unregistered [ 609.223615] Key type ._llcrypt unregistered [ 614.861623] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 614.963273] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 614.983510] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 614.996043] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 617.573510] Key type ._llcrypt registered [ 617.575975] Key type .llcrypt registered [ 617.770290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 617.794172] alg: No test for adler32 (adler32-zlib) [ 618.756339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.798658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 618.825460] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 619.413734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 619.439854] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 621.279227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 621.303728] LNet: Removed LNI 10.1.2.3@tcp [ 621.430406] LNet: 34285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.442279] LNetError: 34285:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.458600] LNet: Removed LNI 192.168.203.49@tcp [ 622.113697] Key type .llcrypt unregistered [ 622.115057] Key type ._llcrypt unregistered [ 632.397400] Key type ._llcrypt registered [ 632.399552] Key type .llcrypt registered [ 633.523754] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 17:35:20 (1763332520) [ 634.279568] Key type .llcrypt unregistered [ 634.281290] Key type ._llcrypt unregistered [ 640.362197] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 640.613451] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 640.632056] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 640.654897] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 643.275873] Key type ._llcrypt registered [ 643.277182] Key type .llcrypt registered [ 643.475354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 643.492794] alg: No test for adler32 (adler32-zlib) [ 644.453770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.521915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 644.571509] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 645.335372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 645.349656] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 647.891529] LNet: 35772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.898421] LNetError: 35772:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.911928] LNet: Removed LNI 192.168.203.49@tcp [ 647.922780] LNet: Removed LNI 10.1.2.3@tcp [ 648.711485] Key type .llcrypt unregistered [ 648.713548] Key type ._llcrypt unregistered [ 659.852733] Key type ._llcrypt registered [ 659.860372] Key type .llcrypt registered [ 661.129452] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 17:35:48 (1763332548) [ 661.512464] Key type .llcrypt unregistered [ 661.514360] Key type ._llcrypt unregistered [ 666.346724] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 666.495374] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 666.500323] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 666.536439] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 669.131956] Key type ._llcrypt registered [ 669.139581] Key type .llcrypt registered [ 669.360172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 669.374926] alg: No test for adler32 (adler32-zlib) [ 670.298290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.362068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 670.394729] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 672.100749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 672.114085] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 673.889883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 673.908780] LNet: Removed LNI 10.1.2.3@tcp [ 673.920655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 673.944769] LNet: Removed LNI 10.1.2.31@tcp [ 673.968747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 674.025057] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 674.811136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 674.848291] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 675.737794] LNet: 37682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.744927] LNetError: 37682:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 675.748451] LNet: Removed LNI 10.1.2.3@tcp [ 675.770429] LNet: Removed LNI 10.1.2.31@tcp [ 676.479491] Key type .llcrypt unregistered [ 676.481613] Key type ._llcrypt unregistered [ 688.089863] Key type ._llcrypt registered [ 688.091463] Key type .llcrypt registered [ 689.339377] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 17:36:16 (1763332576) [ 689.865429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 689.886143] alg: No test for adler32 (adler32-zlib) [ 690.837835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.900753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 690.923517] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 691.734671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.752346] LNet: 38632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.762650] LNetError: 38632:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.776661] LNet: Removed LNI 192.168.203.49@tcp [ 691.807732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.899309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.49@tcp [ 691.931779] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 692.767382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.49@tcp [ 692.818188] LNet: Removed LNI 192.168.203.49@tcp [ 692.906566] LNet: 38767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.571548] Key type .llcrypt unregistered [ 693.578688] Key type ._llcrypt unregistered [ 703.217415] Key type ._llcrypt registered [ 703.219464] Key type .llcrypt registered [ 704.344808] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 17:36:31 (1763332591) [ 706.018945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg349-server [ 707.769834] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 17:36:35 (1763332595) [ 708.240185] Key type .llcrypt unregistered [ 708.246514] Key type ._llcrypt unregistered [ 713.278646] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 713.419858] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 713.421710] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 713.436759] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 716.012545] Key type ._llcrypt registered [ 716.014200] Key type .llcrypt registered [ 716.193401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 716.205541] alg: No test for adler32 (adler32-zlib) [ 717.255551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.339276] LNet: Added LNI 192.168.203.49@tcp1 [8/256/0/180] [ 718.074960] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 720.083819] LNet: 40593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.088942] LNetError: 40593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 720.093071] LNet: Removed LNI 192.168.203.49@tcp1 [ 720.102883] LNet: Removed LNI 10.1.2.3@tcp2 [ 720.665931] Key type .llcrypt unregistered [ 720.667678] Key type ._llcrypt unregistered [ 729.741736] Key type ._llcrypt registered [ 729.743400] Key type .llcrypt registered [ 730.994946] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 17:36:58 (1763332618) [ 731.390447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 731.414389] alg: No test for adler32 (adler32-zlib) [ 732.350319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.416278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 732.440552] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 733.163879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 733.189809] LNet: 41560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.195591] LNetError: 41560:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 733.208456] LNet: Removed LNI 192.168.203.49@tcp [ 733.235191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.285256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 733.312367] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 738.848086] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 17:37:06 (1763332626) [ 739.160451] LNet: 42151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.173323] LNetError: 42151:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 739.189136] LNet: Removed LNI 192.168.203.49@tcp [ 739.729764] Key type .llcrypt unregistered [ 739.734814] Key type ._llcrypt unregistered [ 744.941039] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 745.091466] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 745.093809] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 745.096439] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 747.566480] Key type ._llcrypt registered [ 747.569050] Key type .llcrypt registered [ 747.795461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 747.806496] alg: No test for adler32 (adler32-zlib) [ 748.771776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.814734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 748.859345] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 749.514959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 749.546362] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 751.600308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 751.615234] LNet: 42886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.623937] LNetError: 42886:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 751.627593] LNet: Removed LNI 192.168.203.49@tcp [ 751.637786] LNet: Removed LNI 10.1.2.3@tcp [ 751.661398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.735894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 751.778855] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 751.785624] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 752.353210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.381312] LNet: 42914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.391176] LNetError: 42914:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 752.397534] LNetError: 42914:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 752.414166] LNet: Removed LNI 192.168.203.49@tcp [ 752.429162] LNet: Removed LNI 10.1.2.3@tcp [ 752.455769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.544216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 752.572886] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 752.583925] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 753.102920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 753.124811] LNet: Added LNI 192.168.203.49@tcp2 [8/256/0/180] [ 753.977596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.993742] LNet: 43076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.000181] LNetError: 43076:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 754.009856] LNetError: 43076:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 754.016759] LNet: Removed LNI 192.168.203.49@tcp [ 754.023826] LNet: Removed LNI 10.1.2.3@tcp [ 754.034304] LNet: Removed LNI 192.168.203.49@tcp2 [ 754.066470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.150421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 754.166023] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 754.168146] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 754.171048] LNet: Added LNI 192.168.203.49@tcp2 [8/256/0/180] [ 760.886230] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 17:37:28 (1763332648) [ 761.042986] LNet: 43666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.048092] LNetError: 43666:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 761.055495] LNetError: 43666:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 761.070647] LNet: Removed LNI 192.168.203.49@tcp [ 761.085990] LNet: Removed LNI 10.1.2.3@tcp [ 761.100164] LNet: Removed LNI 192.168.203.49@tcp2 [ 761.777562] Key type .llcrypt unregistered [ 761.779364] Key type ._llcrypt unregistered [ 767.445485] Key type ._llcrypt registered [ 767.447126] Key type .llcrypt registered [ 767.621280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 767.632377] alg: No test for adler32 (adler32-zlib) [ 768.608292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.675160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 768.716520] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 769.600701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 769.623777] LNet: 44065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.632216] LNetError: 44065:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.641898] LNet: Removed LNI 192.168.203.49@tcp [ 769.676572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.778631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 769.809448] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 770.374804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.392945] LNet: 44095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.398659] LNetError: 44095:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.411895] LNet: Removed LNI 192.168.203.49@tcp [ 770.448116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.497597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 770.533741] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 770.970536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.999371] LNet: Removed LNI 192.168.203.49@tcp [ 771.025636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.104513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 771.141964] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 771.610628] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 771.781383] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 771.796251] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 771.803686] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 774.228304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 774.264602] LNet: 44204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.282904] LNet: 44204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 774.305753] LNetError: 44204:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.316230] LNetError: 44204:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 774.337406] LNet: Removed LNI 192.168.203.49@tcp [ 774.368820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.491439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 774.541192] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 781.246559] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 17:37:48 (1763332668) [ 781.527316] LNet: 44799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.535742] LNetError: 44799:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.580977] LNet: Removed LNI 192.168.203.49@tcp [ 782.596209] Key type .llcrypt unregistered [ 782.598587] Key type ._llcrypt unregistered [ 789.043897] Key type ._llcrypt registered [ 789.047897] Key type .llcrypt registered [ 789.296488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 789.314130] alg: No test for adler32 (adler32-zlib) [ 790.299282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.392912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 790.445457] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 791.371891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.389259] LNet: 45194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.394449] LNetError: 45194:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.408347] LNet: Removed LNI 192.168.203.49@tcp [ 791.429479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.489125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 791.543518] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 799.035508] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 17:38:06 (1763332686) [ 799.232807] LNet: 45786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.244224] LNetError: 45786:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.289054] LNet: Removed LNI 192.168.203.49@tcp [ 799.890557] Key type .llcrypt unregistered [ 799.891915] Key type ._llcrypt unregistered [ 805.942432] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 806.155485] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 806.170410] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 806.174882] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 808.849151] Key type ._llcrypt registered [ 808.850733] Key type .llcrypt registered [ 809.098193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 809.124875] alg: No test for adler32 (adler32-zlib) [ 810.097487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.183953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 810.232050] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 811.204222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.216512] LNet: 46231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.222697] LNetError: 46231:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.241180] LNet: Removed LNI 192.168.203.49@tcp [ 811.274892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.378677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 811.401634] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 817.666536] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 17:38:25 (1763332705) [ 817.787730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.813223] LNet: 46821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.816462] LNetError: 46821:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.845570] LNet: Removed LNI 192.168.203.49@tcp [ 817.886169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.081742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 825.029862] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 17:38:32 (1763332712) [ 825.149585] LNet: 47404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.008549] Key type .llcrypt unregistered [ 826.016133] Key type ._llcrypt unregistered [ 831.986465] Key type ._llcrypt registered [ 831.993308] Key type .llcrypt registered [ 832.147531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 832.162766] alg: No test for adler32 (adler32-zlib) [ 833.118412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.186650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 833.235470] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 833.951620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 833.986771] LNet: 47805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.004102] LNetError: 47805:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.025622] LNet: Removed LNI 192.168.203.49@tcp [ 834.052259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.119313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 834.162991] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 840.363915] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 17:38:47 (1763332727) [ 840.496478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.521235] LNet: 48396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.529800] LNetError: 48396:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.549883] LNet: Removed LNI 192.168.203.49@tcp [ 840.579731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.687973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 846.612388] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 17:38:53 (1763332733) [ 848.364872] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 850.011336] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 17:38:57 (1763332737) [ 851.557248] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 853.599105] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 17:39:00 (1763332740) [ 855.157659] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 857.121621] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 17:39:04 (1763332744) [ 858.462043] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 859.930723] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 17:39:07 (1763332747) [ 861.222165] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 862.869917] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 17:39:10 (1763332750) [ 862.996875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.057248] LNet: 50449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.092986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.131618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 863.151712] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 863.788520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.49 [ 863.809884] LNet: Removed LNI 192.168.203.49@tcp [ 869.493888] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 17:39:16 (1763332756) [ 869.674662] LNet: 51154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.681440] LNetError: 51154:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.692263] LNet: Removed LNI 192.168.203.49@tcp [ 870.234666] Key type .llcrypt unregistered [ 870.244312] Key type ._llcrypt unregistered [ 875.009659] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 875.265746] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 875.272844] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 875.283990] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 877.982027] Key type ._llcrypt registered [ 877.989364] Key type .llcrypt registered [ 878.232876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 878.240944] alg: No test for adler32 (adler32-zlib) [ 879.772682] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 885.434349] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 17:39:32 (1763332772) [ 885.544219] LNet: 52047:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.548185] LNetError: 52047:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 885.559975] LNet: Removed LNI 10.1.2.3@tcp [ 886.143911] Key type .llcrypt unregistered [ 886.145575] Key type ._llcrypt unregistered [ 891.219826] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 891.395677] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 891.400617] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 891.406263] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 894.052677] Key type ._llcrypt registered [ 894.057563] Key type .llcrypt registered [ 894.201218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 894.209260] alg: No test for adler32 (adler32-zlib) [ 895.111459] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 901.474278] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 17:39:48 (1763332788) [ 901.600964] LNet: 52938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.614292] LNetError: 52938:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 901.636855] LNet: Removed LNI 10.1.2.3@tcp [ 902.494802] Key type .llcrypt unregistered [ 902.496912] Key type ._llcrypt unregistered [ 907.077117] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 907.288944] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 907.297847] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 907.299940] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 910.060300] Key type ._llcrypt registered [ 910.066657] Key type .llcrypt registered [ 910.251389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 910.263164] alg: No test for adler32 (adler32-zlib) [ 911.303310] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 918.574641] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 17:40:05 (1763332805) [ 918.886436] LNet: 53832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.902675] LNetError: 53832:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 918.922752] LNet: Removed LNI 10.1.2.3@tcp [ 919.641204] Key type .llcrypt unregistered [ 919.642928] Key type ._llcrypt unregistered [ 925.161701] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 925.434624] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 925.437894] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 925.440202] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 928.279462] Key type ._llcrypt registered [ 928.281035] Key type .llcrypt registered [ 928.569047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 928.585613] alg: No test for adler32 (adler32-zlib) [ 929.489563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.568078] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 938.599466] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 17:40:25 (1763332825) [ 939.212696] LNet: 55058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.221560] LNetError: 55058:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 939.239564] LNet: Removed LNI 10.1.2.3@tcp [ 940.001936] Key type .llcrypt unregistered [ 940.003761] Key type ._llcrypt unregistered [ 945.621899] Key type ._llcrypt registered [ 945.627451] Key type .llcrypt registered [ 945.853883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 945.873796] alg: No test for adler32 (adler32-zlib) [ 946.797854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 946.823163] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 953.739354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.149@tcp [ 954.545666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_interrupt -r 1 [ 954.601163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_interrupt -r 1 [ 954.663953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 954.684967] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 954.983412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 955.028821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 955.087497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 955.103996] LNet: 1 local NIs in recovery (showing 1): 192.168.203.49@tcp [ 955.169386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 955.210654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 955.272809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 955.344403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 955.421236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 955.454981] LNet: 55492:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 955.480827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 955.556904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 955.613882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 955.689762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 955.756922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 955.850678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 955.937374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 956.660922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_dropped -r 1 [ 956.697104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_dropped -r 1 [ 956.738873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 956.759686] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 956.763393] LNet: Skipped 1 previous similar message [ 957.026647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 957.065093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 957.115135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 957.165534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 957.208818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 957.255530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 957.298389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 957.342521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 957.362539] LNet: 55602:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 957.382532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 957.418971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 957.462652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 957.500571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 957.552198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 957.578283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 957.618830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 958.182315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_aborted -r 1 [ 958.219452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_aborted -r 1 [ 958.259701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 958.284694] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 958.521780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 958.549930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 958.565796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 958.584623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 958.613792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 958.648292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 958.682636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 958.708972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 958.725874] LNet: 55711:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 958.748332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 958.777466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 958.818755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 958.854710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 958.882731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 958.906562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 958.946325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 959.397252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_no_route -r 1 [ 959.466767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_no_route -r 1 [ 959.510679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 959.708792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 959.735603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 959.776980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 959.814606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 959.841630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 959.888663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 959.944356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 959.991077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 960.046290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 960.095927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 960.136348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 960.182575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 960.221675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 960.257812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 960.295740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 960.784050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_timeout -r 1 [ 960.802606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_timeout -r 1 [ 960.835554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 960.850506] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 960.854436] LNet: Skipped 1 previous similar message [ 961.083799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 961.114408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 961.147502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 961.188310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 961.228378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 961.286262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 961.341199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 961.375631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 961.425053] LNet: 55929:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 961.431176] LNet: 55929:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 961.447370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 961.510912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 961.536733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 961.564477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 961.595420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 961.624932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 961.673368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 962.306335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_error -r 1 [ 962.337957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_error -r 1 [ 962.373278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 962.647872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 962.730441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 962.769677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 962.814594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 962.862711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 962.897625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 962.950711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 962.975605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 963.004664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 963.036988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 963.066617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 963.091493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 963.116487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 963.147803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 963.168903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 963.521269] LNet: 56079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.530761] LNetError: 56079:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.588132] LNet: Removed LNI 192.168.203.49@tcp [ 965.137661] Key type .llcrypt unregistered [ 965.139536] Key type ._llcrypt unregistered [ 981.360145] Key type ._llcrypt registered [ 981.361666] Key type .llcrypt registered [ 982.907316] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 17:41:09 (1763332869) [ 983.860440] Key type .llcrypt unregistered [ 983.863523] Key type ._llcrypt unregistered [ 991.373311] Key type ._llcrypt registered [ 991.374868] Key type .llcrypt registered [ 991.696599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 991.731644] alg: No test for adler32 (adler32-zlib) [ 992.688589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 992.736920] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1000.865500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.149@tcp [ 1011.324609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.387945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1011.453398] LNet: Added LNI 192.168.203.49@tcp1 [8/256/0/180] [ 1013.449182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_interrupt -r 1 [ 1013.498641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e local_interrupt -r 1 [ 1013.559506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_interrupt -r 1 [ 1013.612864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e local_interrupt -r 1 [ 1013.680711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e local_interrupt -r 1 [ 1013.725355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e local_interrupt -r 1 [ 1013.793288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e local_interrupt -r 1 [ 1013.855272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e local_interrupt -r 1 [ 1013.914664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 1013.940758] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1013.947752] LNet: 1 local NIs in recovery (showing 1): 192.168.203.49@tcp [ 1014.239033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1014.271659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1014.295662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1014.344425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1014.381154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1014.401824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1014.443710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1014.482939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1014.506731] LNet: 57542:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1014.538539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1014.565520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1014.593200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1014.616061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1014.654808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1014.694425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1014.728257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1015.362958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_dropped -r 1 [ 1015.408786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e local_dropped -r 1 [ 1015.455751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_dropped -r 1 [ 1015.517256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e local_dropped -r 1 [ 1015.564379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e local_dropped -r 1 [ 1015.603476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e local_dropped -r 1 [ 1015.642304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e local_dropped -r 1 [ 1015.682497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e local_dropped -r 1 [ 1015.728712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 1015.754739] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1015.758641] LNet: Skipped 2 previous similar messages [ 1015.971923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1016.008453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1016.047200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1016.092388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1016.127621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1016.171148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1016.213225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1016.250205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1016.266732] LNet: 57663:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1016.278743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1016.315938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1016.338897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1016.363983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1016.395857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1016.429973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1016.479640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1016.936240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_aborted -r 1 [ 1016.965949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e local_aborted -r 1 [ 1017.009565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_aborted -r 1 [ 1017.086270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e local_aborted -r 1 [ 1017.128385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e local_aborted -r 1 [ 1017.160827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e local_aborted -r 1 [ 1017.182284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e local_aborted -r 1 [ 1017.230294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e local_aborted -r 1 [ 1017.287535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 1017.317836] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1017.330023] LNet: Skipped 2 previous similar messages [ 1017.519609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1017.548138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1017.567453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1017.598379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1017.633831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1017.666702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1017.692857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1017.716151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1017.740454] LNet: 57784:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1017.771805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1017.796698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1017.824238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1017.861440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1017.888972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1017.922518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1017.964586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1018.438793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_no_route -r 1 [ 1018.498840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e local_no_route -r 1 [ 1018.532325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_no_route -r 1 [ 1018.572766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e local_no_route -r 1 [ 1018.599982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e local_no_route -r 1 [ 1018.642801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e local_no_route -r 1 [ 1018.683961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e local_no_route -r 1 [ 1018.736806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e local_no_route -r 1 [ 1018.799662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 1019.043918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1019.088266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1019.127886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1019.155362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1019.186448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1019.214404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1019.251883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1019.277194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1019.328508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1019.362725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1019.403160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1019.434797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1019.469636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1019.505239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.528549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1020.088279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_timeout -r 1 [ 1020.113657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e local_timeout -r 1 [ 1020.143613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_timeout -r 1 [ 1020.178792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e local_timeout -r 1 [ 1020.206899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e local_timeout -r 1 [ 1020.239413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e local_timeout -r 1 [ 1020.283258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e local_timeout -r 1 [ 1020.321349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e local_timeout -r 1 [ 1020.362494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 1020.377932] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1020.381647] LNet: Skipped 5 previous similar messages [ 1020.556562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1020.584451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1020.613551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1020.649560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1020.680518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1020.722201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1020.748391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1020.791950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1020.822637] LNet: 58027:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1020.827674] LNet: 58027:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 1020.843154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1020.875490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1020.916529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1020.940233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1020.972772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1021.013261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1021.043534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1021.512331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_error -r 1 [ 1021.555942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e local_error -r 1 [ 1021.596533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_error -r 1 [ 1021.634410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e local_error -r 1 [ 1021.666515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e local_error -r 1 [ 1021.718279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e local_error -r 1 [ 1021.778312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e local_error -r 1 [ 1021.822254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e local_error -r 1 [ 1021.850376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 1022.035691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1022.067618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1022.104719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1022.141399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1022.187628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1022.232510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1022.278753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1022.336152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1022.388628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1022.450456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1022.503435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1022.553746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1022.590603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1022.631507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1022.667388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1027.993460] LNet: 58223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.006226] LNetError: 58223:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.017291] LNet: Removed LNI 192.168.203.49@tcp [ 1029.030666] LNet: Removed LNI 192.168.203.49@tcp1 [ 1029.504784] Key type .llcrypt unregistered [ 1029.506473] Key type ._llcrypt unregistered [ 1044.835413] Key type ._llcrypt registered [ 1044.836987] Key type .llcrypt registered [ 1046.320260] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 17:42:13 (1763332933) [ 1047.107592] Key type .llcrypt unregistered [ 1047.109726] Key type ._llcrypt unregistered [ 1052.243781] Key type ._llcrypt registered [ 1052.245225] Key type .llcrypt registered [ 1052.559294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1052.590308] alg: No test for adler32 (adler32-zlib) [ 1053.502821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1053.541789] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1060.341272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.149@tcp [ 1061.018390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e remote_dropped -r 1 [ 1061.085898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e remote_dropped -r 1 [ 1061.121234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 1061.144930] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1061.376313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1061.397411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1061.450063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1061.497559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1061.548670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1061.617240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1061.697496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1061.744640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1061.778191] LNet: 59468:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1061.829121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1061.917779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1062.032408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1062.069194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1062.108586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1062.138996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1062.195726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1062.932791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e remote_error -r 1 [ 1062.960353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e remote_error -r 1 [ 1062.994411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 1063.012682] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1063.180538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1063.213563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1063.239210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1063.272561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1063.310590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1063.355479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1063.388845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1063.419335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1063.431890] LNet: 59577:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1063.456875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1063.491692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1063.569217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1063.624297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1063.684446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1063.740877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1063.798669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1064.357629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e remote_timeout -r 1 [ 1064.429760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e remote_timeout -r 1 [ 1064.481547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 1064.531114] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1064.794125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1064.821761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1064.867395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1064.898962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1064.921979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1064.948659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1064.991910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1065.039955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1065.099222] LNet: 59687:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1065.130895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1065.153350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1065.215811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1065.257031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1065.309651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1065.351788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1065.380679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1065.597934] LNet: 59727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.604245] LNetError: 59727:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1065.619691] LNet: Removed LNI 192.168.203.49@tcp [ 1066.020490] Key type .llcrypt unregistered [ 1066.024234] Key type ._llcrypt unregistered [ 1080.369631] Key type ._llcrypt registered [ 1080.383615] Key type .llcrypt registered [ 1081.580969] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 17:42:48 (1763332968) [ 1082.376307] Key type .llcrypt unregistered [ 1082.380286] Key type ._llcrypt unregistered [ 1087.418535] Key type ._llcrypt registered [ 1087.426141] Key type .llcrypt registered [ 1087.738827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1087.757476] alg: No test for adler32 (adler32-zlib) [ 1088.660190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1088.701287] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1095.352580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.149@tcp [ 1103.110876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1103.182133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1103.226316] LNet: Added LNI 192.168.203.49@tcp1 [8/256/0/180] [ 1104.958597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e remote_dropped -r 1 [ 1104.979757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e remote_dropped -r 1 [ 1105.022081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e remote_dropped -r 1 [ 1105.059886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e remote_dropped -r 1 [ 1105.100268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e remote_dropped -r 1 [ 1105.125235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e remote_dropped -r 1 [ 1105.169467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e remote_dropped -r 1 [ 1105.211623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e remote_dropped -r 1 [ 1105.234958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 1105.254706] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1105.261459] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.149@tcp [ 1105.492971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.519445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1105.549119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.589676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1105.619654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.643437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.671493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.690467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.703731] LNet: 61190:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.715272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.738814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.762313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.792670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.841941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.886667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.927840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1105.973067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.465919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e remote_error -r 1 [ 1106.529657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e remote_error -r 1 [ 1106.587149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e remote_error -r 1 [ 1106.635791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e remote_error -r 1 [ 1106.697163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e remote_error -r 1 [ 1106.748050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e remote_error -r 1 [ 1106.801150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e remote_error -r 1 [ 1106.847869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e remote_error -r 1 [ 1106.899288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 1106.937125] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1106.941474] LNet: Skipped 2 previous similar messages [ 1107.208939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.229247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1107.253525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.299935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.337099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.356244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.406490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.440584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.461257] LNet: 61313:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.488226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.531841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.566843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.619282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.653964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.690454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.729277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1107.751512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.365674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e remote_timeout -r 1 [ 1108.403170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e remote_timeout -r 1 [ 1108.455616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e remote_timeout -r 1 [ 1108.523537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e remote_timeout -r 1 [ 1108.576871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e remote_timeout -r 1 [ 1108.636645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e remote_timeout -r 1 [ 1108.676708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e remote_timeout -r 1 [ 1108.713162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e remote_timeout -r 1 [ 1108.740893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.149@tcp [ 1108.756419] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1108.759054] LNet: Skipped 1 previous similar message [ 1108.976745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.003651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1109.031763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.061652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.104765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.136212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.184582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.225634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.245680] LNet: 61436:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1109.280384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.315313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.339027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.377483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.461195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.536586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.601027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1109.667339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.336626] LNet: 61513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.348852] LNetError: 61513:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.359176] LNet: Removed LNI 192.168.203.49@tcp [ 1115.361156] LNetError: 61513:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.396557] LNet: Removed LNI 192.168.203.49@tcp1 [ 1115.866145] Key type .llcrypt unregistered [ 1115.867910] Key type ._llcrypt unregistered [ 1132.809396] Key type ._llcrypt registered [ 1132.811073] Key type .llcrypt registered [ 1134.074385] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 17:43:41 (1763333021) [ 1134.522131] Key type .llcrypt unregistered [ 1134.525289] Key type ._llcrypt unregistered [ 1139.940632] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1140.080885] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1140.089055] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1140.103597] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1142.589134] Key type ._llcrypt registered [ 1142.604406] Key type .llcrypt registered [ 1142.858817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1142.869549] alg: No test for adler32 (adler32-zlib) [ 1143.823059] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1143.844382] LNet: 62599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.848456] LNetError: 62599:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1143.862648] LNet: Removed LNI 192.168.203.49@tcp [ 1144.553388] Key type .llcrypt unregistered [ 1144.555214] Key type ._llcrypt unregistered [ 1150.565547] Key type ._llcrypt registered [ 1150.580562] Key type .llcrypt registered [ 1150.747146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1150.771029] alg: No test for adler32 (adler32-zlib) [ 1151.720258] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1151.727331] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1151.767992] LNet: 62868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.773689] LNetError: 62868:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.783141] LNet: Removed LNI 192.168.203.49@tcp [ 1151.801468] LNet: Removed LNI 10.1.2.3@tcp1 [ 1152.519439] Key type .llcrypt unregistered [ 1152.520777] Key type ._llcrypt unregistered [ 1158.728514] Key type ._llcrypt registered [ 1158.729884] Key type .llcrypt registered [ 1158.988873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1159.016649] alg: No test for adler32 (adler32-zlib) [ 1160.062973] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1160.066307] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1160.137427] LNet: 63137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1160.146368] LNetError: 63137:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1160.153292] LNet: Removed LNI 192.168.203.49@tcp [ 1160.164870] LNet: Removed LNI 10.1.2.3@tcp [ 1160.928418] Key type .llcrypt unregistered [ 1160.932242] Key type ._llcrypt unregistered [ 1167.452596] Key type ._llcrypt registered [ 1167.454208] Key type .llcrypt registered [ 1167.710836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1167.759313] alg: No test for adler32 (adler32-zlib) [ 1168.718095] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1168.720619] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1168.782907] LNet: 63408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.787291] LNetError: 63408:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.795766] LNet: Removed LNI 192.168.203.49@tcp [ 1168.803710] LNet: Removed LNI 10.1.2.3@tcp [ 1169.590531] Key type .llcrypt unregistered [ 1169.592360] Key type ._llcrypt unregistered [ 1175.586764] Key type ._llcrypt registered [ 1175.594821] Key type .llcrypt registered [ 1175.941343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1175.972889] alg: No test for adler32 (adler32-zlib) [ 1176.960962] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1176.964867] LNetError: 63669:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1176.970493] LNetError: Error -100 starting up LNI tcp [ 1176.981758] LNet: Removed LNI 192.168.203.49@tcp [ 1177.567705] Key type .llcrypt unregistered [ 1177.576372] Key type ._llcrypt unregistered [ 1188.557485] Key type ._llcrypt registered [ 1188.559190] Key type .llcrypt registered [ 1190.031535] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 17:44:37 (1763333077) [ 1190.864678] Key type .llcrypt unregistered [ 1190.867328] Key type ._llcrypt unregistered [ 1195.871277] Key type ._llcrypt registered [ 1195.872957] Key type .llcrypt registered [ 1196.083723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1196.098110] alg: No test for adler32 (adler32-zlib) [ 1197.043112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1197.067603] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1204.018738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.149@tcp [ 1204.793648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e network_timeout -r 1 [ 1204.822816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e network_timeout -r 1 [ 1204.854153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.149@tcp [ 1204.877587] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1205.114226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1205.141217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1205.168962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1205.190821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1205.226516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1205.258913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1205.284990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1205.294312] LNet: 1 local NIs in recovery (showing 1): 192.168.203.49@tcp [ 1205.322235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1205.340344] LNet: 64918:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1205.357312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1205.387815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1205.416698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1205.433118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1205.449376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.468336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1205.654935] LNet: 64956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1205.661145] LNetError: 64956:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1205.688203] LNet: Removed LNI 192.168.203.49@tcp [ 1206.059941] Key type .llcrypt unregistered [ 1206.061646] Key type ._llcrypt unregistered [ 1221.792602] Key type ._llcrypt registered [ 1221.794766] Key type .llcrypt registered [ 1222.022546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1222.066390] alg: No test for adler32 (adler32-zlib) [ 1223.057639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1223.098736] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1229.610937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.149@tcp [ 1236.750875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1236.779903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1236.799498] LNet: Added LNI 192.168.203.49@tcp1 [8/256/0/180] [ 1238.050308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e network_timeout -r 1 [ 1238.089414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e network_timeout -r 1 [ 1238.116651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e network_timeout -r 1 [ 1238.138375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e network_timeout -r 1 [ 1238.164200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e network_timeout -r 1 [ 1238.192071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e network_timeout -r 1 [ 1238.216313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e network_timeout -r 1 [ 1238.240772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e network_timeout -r 1 [ 1238.278152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.149@tcp [ 1238.300397] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1238.504987] LNet: 1 local NIs in recovery (showing 1): 192.168.203.49@tcp [ 1238.515866] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.149@tcp [ 1238.543343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1238.579910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1238.635392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1238.666106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1238.690342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1238.710224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1238.734911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1238.757737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1238.774287] LNet: 65829:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1238.791806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1238.829547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1238.871650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1238.898252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.928102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.975936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1242.823739] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1243.184687] LNet: 65902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.190432] LNetError: 65902:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1244.193315] LNet: Removed LNI 192.168.203.49@tcp [ 1244.196574] LNetError: 65902:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1244.219902] LNet: Removed LNI 192.168.203.49@tcp1 [ 1244.607664] Key type .llcrypt unregistered [ 1244.613766] Key type ._llcrypt unregistered [ 1258.779357] Key type ._llcrypt registered [ 1258.780906] Key type .llcrypt registered [ 1259.811665] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 17:45:47 (1763333147) [ 1260.304545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1260.323942] alg: No test for adler32 (adler32-zlib) [ 1261.235356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1261.294919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1261.319335] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1262.151702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1262.171379] LNet: Added LNI 192.168.203.49@tcp1 [8/256/0/180] [ 1262.968243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 1263.032682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1263.076206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1263.142593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1263.187219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1263.214956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1263.250355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.49@tcp [ 1263.282814] LNetError: 66999:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.203.49@tcp to recovery [ 1263.329745] LNet: 1 local NIs in recovery (showing 1): 192.168.203.49@tcp [ 1263.336778] LNet: There was an unexpected network error while writing to 192.168.203.49: rc = -22 [ 1265.375334] LNet: There was an unexpected network error while writing to 192.168.203.49: rc = -22 [ 1265.633766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1269.471562] LNet: There was an unexpected network error while writing to 192.168.203.49: rc = -22 [ 1270.293411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1270.361042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1270.456675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1270.484916] LNet: 67106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1270.498407] LNetError: 67106:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1270.507148] LNet: Removed LNI 192.168.203.49@tcp [ 1270.527706] LNet: Removed LNI 192.168.203.49@tcp1 [ 1270.570725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1270.627575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1270.671480] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1271.377764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1271.396537] LNet: Added LNI 192.168.203.49@tcp1 [8/256/0/180] [ 1272.121878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 1272.158965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1272.222576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1272.282109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1272.345554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1272.399583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1272.445300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1272.473862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.49@tcp [ 1272.487451] LNetError: 67378:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.203.49@tcp to recovery [ 1272.671570] LNet: There was an unexpected network error while writing to 192.168.203.49: rc = -22 [ 1275.945138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1278.818542] LNet: There was an unexpected network error while writing to 192.168.203.49: rc = -22 [ 1278.831257] LNet: Skipped 1 previous similar message [ 1283.967589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1284.029050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1284.087915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1290.120787] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 17:46:17 (1763333177) [ 1290.227356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1290.243826] LNet: 68087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.248513] LNetError: 68087:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1290.253142] LNetError: 68087:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1290.257876] LNet: Removed LNI 192.168.203.49@tcp [ 1290.268375] LNet: Removed LNI 192.168.203.49@tcp1 [ 1290.282379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1290.337767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1290.353758] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1291.008096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1291.026891] LNet: Added LNI 192.168.203.49@tcp1 [8/256/0/180] [ 1291.882858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 1291.948447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1291.978477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1292.010789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1292.044746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1292.124314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1292.164923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.49@tcp --health 0 [ 1292.274554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1292.386496] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.49@tcp [ 1293.408163] LNet: There was an unexpected network error while writing to 192.168.203.49: rc = -22 [ 1293.423180] LNet: Skipped 1 previous similar message [ 1308.517924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1308.682203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1308.708714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1308.737129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.49@tcp --health 500 [ 1308.850522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1315.443408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1315.555415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1315.597687] LNet: 68595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1315.615203] LNetError: 68595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1315.618437] LNetError: 68595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1316.639739] LNet: Removed LNI 192.168.203.49@tcp [ 1316.666293] LNet: Removed LNI 192.168.203.49@tcp1 [ 1316.711227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1316.805429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1316.866991] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1317.555817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1317.582721] LNet: Added LNI 192.168.203.49@tcp1 [8/256/0/180] [ 1318.475734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 1318.501334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1318.555880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1318.585460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1318.618719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1318.661721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1318.696777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1318.724785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.49@tcp --health 0 [ 1319.867704] LNet: There was an unexpected network error while writing to 192.168.203.49: rc = -22 [ 1319.870902] LNet: Skipped 3 previous similar messages [ 1321.136474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1330.061857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1330.107561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1330.144077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1336.207192] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 17:47:03 (1763333223) [ 1336.730494] LNet: 69640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.735765] LNetError: 69640:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.740151] LNetError: 69640:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1336.745312] LNet: Removed LNI 192.168.203.49@tcp [ 1336.764691] LNet: Removed LNI 192.168.203.49@tcp1 [ 1337.265384] Key type .llcrypt unregistered [ 1337.270633] Key type ._llcrypt unregistered [ 1341.441263] Key type ._llcrypt registered [ 1341.442740] Key type .llcrypt registered [ 1341.626179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1341.639315] alg: No test for adler32 (adler32-zlib) [ 1342.606627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1342.637451] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1349.206432] Lustre: DEBUG MARKER: Initial discovery [ 1349.360446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.149@tcp [ 1351.199533] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1351.353922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1351.388064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.149@tcp [ 1351.401131] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1351.419955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1351.446534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.149@tcp -r 1 -m GET -l 3 [ 1351.484673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.149@tcp -r 1 -m GET -e local_error [ 1351.511338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.149@tcp -r 1 -m PUT -l 6 [ 1351.903090] LNet: 1 local NIs in recovery (showing 1): 192.168.203.49@tcp [ 1352.564480] Lustre: DEBUG MARKER: Force oleg349-client.virtnet to discover 192.168.203.149@tcp (in background) [ 1352.806134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.149@tcp [ 1353.903739] Lustre: DEBUG MARKER: Force oleg349-server to discover 192.168.203.49@tcp [ 1355.871552] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1356.168493] Lustre: DEBUG MARKER: Wait for 70399 [ 1357.325548] Lustre: DEBUG MARKER: Finished wait on 70399 [ 1357.627282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1357.671625] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1357.685229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1357.972987] LNet: 70846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.984662] LNetError: 70846:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.998731] LNet: Removed LNI 192.168.203.49@tcp [ 1358.379560] Key type .llcrypt unregistered [ 1358.381167] Key type ._llcrypt unregistered [ 1371.283092] Key type ._llcrypt registered [ 1371.284820] Key type .llcrypt registered [ 1372.280223] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 17:47:39 (1763333259) [ 1372.779404] Key type .llcrypt unregistered [ 1372.785386] Key type ._llcrypt unregistered [ 1377.273763] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1377.385898] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1377.388449] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1377.391213] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1379.787734] Key type ._llcrypt registered [ 1379.789677] Key type .llcrypt registered [ 1379.904931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1379.926175] alg: No test for adler32 (adler32-zlib) [ 1380.808886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1380.865221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1380.899562] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1381.478614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1381.491338] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1388.145845] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 17:47:55 (1763333275) [ 1388.329770] LNet: 72917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.338330] LNetError: 72917:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.343384] LNet: Removed LNI 192.168.203.49@tcp [ 1388.359348] LNet: Removed LNI 10.1.2.3@tcp [ 1388.800604] Key type .llcrypt unregistered [ 1388.808489] Key type ._llcrypt unregistered [ 1393.136904] Key type ._llcrypt registered [ 1393.138146] Key type .llcrypt registered [ 1393.368563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1393.380499] alg: No test for adler32 (adler32-zlib) [ 1394.348493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1394.427333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1394.471752] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1395.169875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1395.182371] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1396.066873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.49@tcp [ 1397.152177] LNetError: Unexpected error -22 connecting to 192.168.203.49@tcp at host 192.168.203.49:988 [ 1401.490420] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 17:48:09 (1763333289) [ 1401.628216] LNet: 74042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.637926] LNetError: 74042:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.651812] LNet: Removed LNI 192.168.203.49@tcp [ 1401.665140] LNet: Removed LNI 10.1.2.3@tcp [ 1402.120212] Key type .llcrypt unregistered [ 1402.121814] Key type ._llcrypt unregistered [ 1406.510653] Key type ._llcrypt registered [ 1406.511819] Key type .llcrypt registered [ 1406.692517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1406.722196] alg: No test for adler32 (adler32-zlib) [ 1407.673492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1407.723402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1407.757356] LNet: Added LNI 192.168.203.49@tcp1 [8/256/0/180] [ 1408.437343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1408.465486] LNet: Added LNI 192.168.203.49@tcp2 [8/256/0/180] [ 1409.315978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.49@tcp1 --nid 192.168.203.49@tcp2 [ 1412.685017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.49@tcp1 192.168.203.49@tcp2 [ 1412.705653] LNetError: 74876:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.49@tcp1 to 192.168.203.49@tcp2 - Route table may be misconfigured [ 1412.719556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.49@tcp2 192.168.203.49@tcp1 [ 1417.911080] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 17:48:25 (1763333305) [ 1417.978646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1418.010437] LNet: 75444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1418.013977] LNetError: 75444:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1419.039135] LNet: Removed LNI 192.168.203.49@tcp1 [ 1419.041976] LNetError: 75444:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.076923] LNet: Removed LNI 192.168.203.49@tcp2 [ 1420.116834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.182554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1420.210219] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1420.738542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1420.759181] LNet: Added LNI 192.168.203.49@tcp1 [8/256/0/180] [ 1421.402653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 1421.434613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.49@tcp 192.168.203.49@tcp [ 1421.461835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.49@tcp1 192.168.203.49@tcp1 [ 1421.490213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.49@tcp -d 192.168.203.49@tcp -e network_timeout [ 1421.526147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -e network_timeout [ 1421.562417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -e network_timeout [ 1421.603269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -e network_timeout [ 1421.628072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1421.652566] LNet: There was an unexpected network error while writing to 192.168.203.49: rc = -22 [ 1421.677634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1421.762681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1421.826732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1422.239221] LNet: 1 local NIs in recovery (showing 1): 192.168.203.49@tcp [ 1426.394933] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 17:48:33 (1763333313) [ 1426.549104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1426.584673] LNet: 76312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.590716] LNetError: 76312:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.615195] LNet: Removed LNI 192.168.203.49@tcp [ 1428.646377] LNet: Removed LNI 192.168.203.49@tcp1 [ 1428.683913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1428.809349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1429.120425] LNet: 76348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1429.634815] Key type .llcrypt unregistered [ 1429.636813] Key type ._llcrypt unregistered [ 1439.081891] Key type ._llcrypt registered [ 1439.086689] Key type .llcrypt registered [ 1440.317382] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 17:48:47 (1763333327) [ 1440.675179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1440.685507] alg: No test for adler32 (adler32-zlib) [ 1441.693394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1443.054688] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1444.507908] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 17:48:52 (1763333332) [ 1444.701766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1444.730049] LNet: 77443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1444.770830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1444.844605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1444.882367] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1445.504257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1445.529508] LNet: Added LNI 192.168.203.49@tcp1 [8/256/0/180] [ 1446.358497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1446.394485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp1 [ 1446.429352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp1 [ 1451.680336] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 17:48:59 (1763333339) [ 1453.518464] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg349-server [ 1455.224052] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 17:49:02 (1763333342) [ 1457.056267] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg349-server [ 1458.342350] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 17:49:05 (1763333345) [ 1459.948736] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg349-server [ 1461.480096] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 17:49:09 (1763333349) [ 1463.039478] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg349-server [ 1464.556579] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 17:49:12 (1763333352) [ 1465.924872] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg349-server [ 1467.594385] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 17:49:15 (1763333355) [ 1468.874818] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg349-server [ 1470.213245] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 17:49:17 (1763333357) [ 1471.819772] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg349-server [ 1473.210642] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 17:49:20 (1763333360) [ 1474.796335] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg349-server [ 1476.024093] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 17:49:23 (1763333363) [ 1477.387883] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg349-server [ 1478.999989] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 17:49:26 (1763333366) [ 1479.085640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1479.112688] LNet: 81314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.121188] LNetError: 81314:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1479.148054] LNet: Removed LNI 192.168.203.49@tcp [ 1480.166448] LNet: Removed LNI 192.168.203.49@tcp1 [ 1480.184301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.237227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1480.266321] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1480.891450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1480.950570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1481.132495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1481.158299] LNet: 81467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.165395] LNetError: 81467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.169322] LNetError: 81467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1481.211292] LNet: Removed LNI 192.168.203.49@tcp [ 1481.239688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.290470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1481.324991] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1482.247488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1482.550650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1482.597918] LNetError: 81613:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1483.625272] LNet: Removed LNI 192.168.203.49@tcp [ 1483.659198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.765696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1483.819763] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1484.677848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1484.777138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1484.991982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1485.027724] LNet: 81760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.032882] LNet: 81760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1485.061348] LNetError: 81760:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.119136] LNet: Removed LNI 192.168.203.49@tcp [ 1486.162205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.245962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1486.276963] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1487.337233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1488.731424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1489.773274] LNet: Removed LNI 192.168.203.49@tcp [ 1489.801742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1489.842205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1489.877429] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1490.633949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1490.754118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1491.058415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1491.105682] LNet: 82062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1491.113415] LNet: 82062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1491.124209] LNetError: 82062:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1491.127361] LNetError: 82062:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1492.136622] LNet: Removed LNI 192.168.203.49@tcp [ 1492.178584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.231584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1492.293387] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1493.329791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1493.652264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1494.697865] LNet: Removed LNI 192.168.203.49@tcp [ 1494.726102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.808849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1494.846310] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1495.583667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1495.650848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1495.889013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1497.000084] LNet: Removed LNI 192.168.203.49@tcp [ 1497.041612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1497.120300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1497.156343] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1497.828575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1500.262117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1500.284571] LNet: 82518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.287759] LNet: 82518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1500.293376] LNetError: 82518:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.296450] LNetError: 82518:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1500.314619] LNet: Removed LNI 192.168.203.49@tcp [ 1500.339738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1500.446552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1500.477326] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1501.247625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1501.321526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1501.586377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1502.631850] LNet: Removed LNI 192.168.203.49@tcp [ 1502.684810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1502.768547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1502.812856] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1503.927810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1505.348759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1506.406741] LNet: Removed LNI 192.168.203.49@tcp [ 1506.447843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1506.512309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1506.550995] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1507.419737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1507.505919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1507.845432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1508.901308] LNet: Removed LNI 192.168.203.49@tcp [ 1508.978451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.082923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1509.130838] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1510.106847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1510.494343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.594111] LNet: Removed LNI 192.168.203.49@tcp [ 1511.637680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.687347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1511.727415] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1512.603938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1512.680990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1512.722253] LNet: Removed LNI 192.168.203.49@tcp [ 1512.759423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.840531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1512.873116] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1513.795250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1520.000414] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 17:50:07 (1763333407) [ 1520.094679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1520.129248] LNet: 83942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1520.138831] LNet: 83942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1520.158130] LNetError: 83942:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1520.161602] LNetError: 83942:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1520.187586] LNet: Removed LNI 192.168.203.49@tcp [ 1520.213690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1520.292920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1520.336327] LNet: Added LNI 192.168.203.49@tcp231 [8/256/0/180] [ 1521.845849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1521.890908] LNet: Removed LNI 192.168.203.49@tcp231 [ 1521.916728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.056761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1522.078668] LNet: Added LNI 192.168.203.49@tcp231 [8/256/0/180] [ 1522.088264] LNet: 84095:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1522.739880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1522.776220] LNet: Removed LNI 192.168.203.49@tcp231 [ 1522.787766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1522.814617] LNet: Added LNI 192.168.203.49@tcp231 [8/256/0/180] [ 1523.903115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1523.952453] LNet: Removed LNI 192.168.203.49@tcp231 [ 1523.987079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1524.051642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1524.081648] LNet: Added LNI 192.168.203.49@tcp231 [8/256/0/0] [ 1524.092662] LNet: 84247:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1524.452472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1524.475356] LNet: Removed LNI 192.168.203.49@tcp231 [ 1524.492447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1524.511740] LNet: Added LNI 192.168.203.49@tcp231 [8/256/0/0] [ 1525.500939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1525.542113] LNet: Removed LNI 192.168.203.49@tcp231 [ 1525.559665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.625788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1525.673995] LNet: Added LNI 192.168.203.49@tcp231 [8/256/0/60] [ 1525.676853] LNet: 84399:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1526.053224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1526.087301] LNet: Removed LNI 192.168.203.49@tcp231 [ 1526.096488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1526.129923] LNet: Added LNI 192.168.203.49@tcp231 [8/256/0/60] [ 1527.147521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1527.185225] LNet: Removed LNI 192.168.203.49@tcp231 [ 1527.209076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.272477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1527.303784] LNet: Added LNI 192.168.203.49@tcp231 [8/256/0/180] [ 1527.652780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1527.689831] LNet: Removed LNI 192.168.203.49@tcp231 [ 1527.701486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1527.734486] LNet: Added LNI 192.168.203.49@tcp231 [8/256/0/180] [ 1534.660245] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 17:50:21 (1763333421) [ 1534.746921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.797627] LNet: Removed LNI 192.168.203.49@tcp231 [ 1534.843060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.018866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1535.061543] LNet: Added LNI 192.168.203.49@tcp232 [8/256/0/180] [ 1535.749599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1535.788352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1535.827437] LNet: Removed LNI 192.168.203.49@tcp232 [ 1535.855914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1535.898972] LNet: Added LNI 192.168.203.49@tcp232 [8/256/0/180] [ 1536.904459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1536.938681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1536.983582] LNet: Removed LNI 192.168.203.49@tcp232 [ 1537.000699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1537.034163] LNet: Added LNI 192.168.203.49@tcp232 [8/256/0/180] [ 1537.777694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1537.810905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1537.857142] LNet: Removed LNI 192.168.203.49@tcp232 [ 1537.889423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1537.911107] LNet: Added LNI 192.168.203.49@tcp232 [8/256/0/180] [ 1538.608375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1538.635278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1538.670766] LNet: Removed LNI 192.168.203.49@tcp232 [ 1538.685633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1545.478232] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 17:50:32 (1763333432) [ 1547.020403] Key type .llcrypt unregistered [ 1547.022181] Key type ._llcrypt unregistered [ 1552.466677] Key type ._llcrypt registered [ 1552.468660] Key type .llcrypt registered [ 1552.651818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1552.666633] alg: No test for adler32 (adler32-zlib) [ 1553.555523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1553.595469] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1560.094729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.149@tcp [ 1568.630468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.728932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1568.786271] LNet: Added LNI 192.168.203.49@tcp1 [8/256/0/180] [ 1570.736169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_interrupt -r 2 [ 1570.809115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e local_interrupt -r 2 [ 1570.852711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_interrupt -r 2 [ 1570.900200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e local_interrupt -r 2 [ 1570.936906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e local_interrupt -r 2 [ 1570.998766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e local_interrupt -r 2 [ 1571.061617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e local_interrupt -r 2 [ 1571.134034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e local_interrupt -r 2 [ 1571.193326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.49@tcp 192.168.203.149@tcp [ 1571.260995] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1571.274789] LNet: 1 local NIs in recovery (showing 1): 192.168.203.49@tcp [ 1571.642583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1571.711327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1571.745672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1571.795762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1571.848491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1571.905958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1571.945322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1572.018864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1572.067564] LNet: 87046:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1572.101603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1572.151111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1572.231035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1572.288419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1572.350696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1572.426617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1573.031453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_dropped -r 2 [ 1573.089514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e local_dropped -r 2 [ 1573.173802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_dropped -r 2 [ 1573.220613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e local_dropped -r 2 [ 1573.298875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e local_dropped -r 2 [ 1573.377608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e local_dropped -r 2 [ 1573.430391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e local_dropped -r 2 [ 1573.486639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e local_dropped -r 2 [ 1573.581895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.49@tcp 192.168.203.149@tcp [ 1573.601249] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1573.931449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1573.965935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1573.994815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1574.033642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1574.080328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1574.126237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1574.145830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1574.188885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1574.212295] LNet: 87166:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1574.281975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1574.322259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1574.408041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1574.463994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1574.520109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1574.597519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1575.292891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_aborted -r 2 [ 1575.343189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e local_aborted -r 2 [ 1575.392489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_aborted -r 2 [ 1575.470411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e local_aborted -r 2 [ 1575.526263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e local_aborted -r 2 [ 1575.561714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e local_aborted -r 2 [ 1575.617706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e local_aborted -r 2 [ 1575.685631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e local_aborted -r 2 [ 1575.727554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.49@tcp 192.168.203.149@tcp [ 1575.766495] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1576.007480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1576.028982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1576.063387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1576.095685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1576.139105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1576.176963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1576.206380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1576.229921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1576.249050] LNet: 87286:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1576.275440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1576.352378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1576.402149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1576.467906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1576.539955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1576.608033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1577.327566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_no_route -r 2 [ 1577.375800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e local_no_route -r 2 [ 1577.439108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_no_route -r 2 [ 1577.489991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e local_no_route -r 2 [ 1577.527632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e local_no_route -r 2 [ 1577.552623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e local_no_route -r 2 [ 1577.570988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e local_no_route -r 2 [ 1577.598032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e local_no_route -r 2 [ 1577.626270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.49@tcp 192.168.203.149@tcp [ 1577.892585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1577.937511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1577.984806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1578.024638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1578.075819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1578.134538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1578.189937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1578.236541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1578.254299] LNet: 87407:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1578.285893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1578.328235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1578.371309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1578.404938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1578.437646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1578.462824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1579.050488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e local_timeout -r 2 [ 1579.078435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e local_timeout -r 2 [ 1579.109121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e local_timeout -r 2 [ 1579.137692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e local_timeout -r 2 [ 1579.183192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e local_timeout -r 2 [ 1579.213157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e local_timeout -r 2 [ 1579.275385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e local_timeout -r 2 [ 1579.321442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e local_timeout -r 2 [ 1579.346459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.49@tcp 192.168.203.149@tcp [ 1579.371151] LNet: There was an unexpected network error while writing to 192.168.203.149: rc = -22 [ 1579.381242] LNet: Skipped 1 previous similar message [ 1579.649521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1579.710069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1579.751138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1579.790869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1579.851689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1579.900697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1579.957660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1580.006055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1580.060221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1580.085502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1580.117493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1580.167433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1580.223046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1580.284818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1580.818405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp -m GET -e remote_dropped -r 2 [ 1580.859905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp -d 192.168.203.149@tcp1 -m GET -e remote_dropped -r 2 [ 1580.899799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp -m GET -e remote_dropped -r 2 [ 1580.953511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp -d 192.168.203.49@tcp1 -m GET -e remote_dropped -r 2 [ 1581.013486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp -m GET -e remote_dropped -r 2 [ 1581.066224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.149@tcp1 -m GET -e remote_dropped -r 2 [ 1581.116654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp -m GET -e remote_dropped -r 2 [ 1581.164466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.49@tcp1 -d 192.168.203.49@tcp1 -m GET -e remote_dropped -r 2 [ 1581.195313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.49@tcp 192.168.203.149@tcp [ 1581.207578] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.149@tcp [ 1581.455634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1581.478899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1581.511049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1581.538124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1581.561642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1581.583480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1581.613186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1581.634236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1581.660916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1581.686507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1581.707907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1581.748444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1581.779494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1581.810349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1586.721835] LNet: 87720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.735609] LNetError: 87720:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.742328] LNet: Removed LNI 192.168.203.49@tcp [ 1586.751985] LNet: Removed LNI 192.168.203.49@tcp1 [ 1587.208667] Key type .llcrypt unregistered [ 1587.210451] Key type ._llcrypt unregistered [ 1600.905404] Key type ._llcrypt registered [ 1600.907087] Key type .llcrypt registered [ 1602.161318] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 17:51:29 (1763333489) [ 1602.435433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1602.444632] alg: No test for adler32 (adler32-zlib) [ 1603.410986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.459452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1603.492430] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1604.250403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1604.279826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.49@tcp [ 1604.293975] LNetError: 88695:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.203.49@tcp to recovery [ 1604.337573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.49@tcp [ 1604.402899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1610.631359] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 17:51:37 (1763333497) [ 1610.750242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.799026] LNet: 89277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.811279] LNetError: 89277:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.854160] LNet: Removed LNI 192.168.203.49@tcp [ 1610.891529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.026545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1611.059240] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1611.829731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1611.870724] LNet: Added LNI 192.168.203.49@tcp1 [8/256/0/180] [ 1613.114627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.49@tcp [ 1613.151234] LNetError: 89550:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.203.49@tcp to recovery [ 1614.054442] LNet: 1 local NIs in recovery (showing 1): 192.168.203.49@tcp [ 1614.057633] LNet: There was an unexpected network error while writing to 192.168.203.49: rc = -22 [ 1614.520399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.49@tcp [ 1622.857509] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 17:51:49 (1763333509) [ 1624.580373] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg349-server [ 1626.480897] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 17:51:53 (1763333513) [ 1626.669869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.716581] LNet: 90501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.736049] LNetError: 90501:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.749393] LNet: Removed LNI 192.168.203.49@tcp [ 1626.781719] LNet: Removed LNI 192.168.203.49@tcp1 [ 1626.825338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.885706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1626.913341] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1627.791349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1627.877037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1627.933352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1628.079848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1628.133338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1628.191427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1634.845234] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 17:52:02 (1763333522) [ 1635.163765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.213282] LNet: 91231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1635.223279] LNetError: 91231:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1635.229174] LNetError: 91231:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1635.252410] LNet: Removed LNI 192.168.203.49@tcp [ 1635.293284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.382217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1635.399885] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1642.876272] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 17:52:09 (1763333529) [ 1642.969451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.999884] LNet: 91925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1643.008644] LNetError: 91925:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1643.039715] LNet: Removed LNI 192.168.203.49@tcp [ 1643.079477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.194328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1643.220495] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1650.788043] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 17:52:18 (1763333538) [ 1652.669779] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1654.868661] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 17:52:21 (1763333541) [ 1655.335393] LNet: 92970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.339939] LNetError: 92970:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.374150] LNet: Removed LNI 192.168.203.49@tcp [ 1656.103579] Key type .llcrypt unregistered [ 1656.109762] Key type ._llcrypt unregistered [ 1661.842304] Key type ._llcrypt registered [ 1661.844022] Key type .llcrypt registered [ 1662.042855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1662.051660] alg: No test for adler32 (adler32-zlib) [ 1663.005972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1663.068462] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1670.287189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.149@tcp [ 1674.642535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.149@tcp [ 1674.918743] LNet: 93382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.922870] LNetError: 93382:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.937187] LNet: Removed LNI 192.168.203.49@tcp [ 1675.350445] Key type .llcrypt unregistered [ 1675.352226] Key type ._llcrypt unregistered [ 1684.580448] Key type ._llcrypt registered [ 1684.582066] Key type .llcrypt registered [ 1685.738504] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1687.270068] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1688.900891] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 17:52:56 (1763333576) [ 1689.498708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1689.538272] alg: No test for adler32 (adler32-zlib) [ 1690.498172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.600842] LNet: 94466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.380502] Key type .llcrypt unregistered [ 1691.382183] Key type ._llcrypt unregistered [ 1697.808838] Key type ._llcrypt registered [ 1697.810766] Key type .llcrypt registered [ 1698.049270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1698.104167] alg: No test for adler32 (adler32-zlib) [ 1699.055614] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1699.058699] LNet: 94718:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1699.114168] LNetError: 94730:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.51@tcp is being used as a gateway but routing feature is not turned on [ 1699.130531] LNetError: 94730:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.203.51@tcp has gone from up to down [ 1699.596128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.631117] LNet: 94776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.637743] LNetError: 94776:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1699.666643] LNet: Removed LNI 192.168.203.49@tcp [ 1699.696560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.757329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1699.794776] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1699.802360] LNet: 94783:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1699.815325] LNet: 94783:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1699.818634] LNetError: 94783:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.203.51@tcp is a gateway. Can not delete it [ 1699.838900] LNetError: 94783:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.203.51@tcp is a router. Can not be deleted [ 1699.863948] LNetError: 94780:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.52@tcp is being used as a gateway but routing feature is not turned on [ 1699.882702] LNetError: 94780:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 1699.897646] LNetError: 94780:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.203.52@tcp has gone from up to down [ 1699.906596] LNetError: 94780:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1699.915470] LNet: 94783:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1707.311976] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 17:53:14 (1763333594) [ 1709.282953] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg349-server [ 1711.201189] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 17:53:18 (1763333598) [ 1712.840650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg349-server [ 1714.339927] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 17:53:21 (1763333601) [ 1715.702867] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 1717.591215] Key type lgssc registered [ 1718.701872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1736.290557] Key type lgssc unregistered [ 1736.680484] LNet: 97203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.687383] LNetError: 97203:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.714378] LNet: Removed LNI 192.168.203.49@tcp [ 1737.225663] Key type .llcrypt unregistered [ 1737.227385] Key type ._llcrypt unregistered [ 1751.074710] Key type ._llcrypt registered [ 1751.077782] Key type .llcrypt registered [ 1752.358784] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 17:53:59 (1763333639) [ 1753.093511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1753.116031] alg: No test for adler32 (adler32-zlib) [ 1754.063318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.120610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1754.187486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1754.305562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1754.433659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1760.586899] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 17:54:07 (1763333647) [ 1760.831413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.856805] LNet: 98607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.915075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.954503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1760.973322] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1761.793893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.891497] LNet: 98752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.895345] LNetError: 98752:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.915207] LNet: Removed LNI 192.168.203.49@tcp [ 1761.951769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.016494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1762.064539] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1762.381525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.409974] LNetError: 98780:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.421791] LNet: Removed LNI 192.168.203.49@tcp [ 1762.444274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.480437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1762.528696] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1768.421225] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 17:54:16 (1763333656) [ 1768.662491] LNet: 99371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1768.666859] LNet: 99371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1768.681981] LNetError: 99371:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1768.698125] LNet: Removed LNI 192.168.203.49@tcp [ 1769.242666] Key type .llcrypt unregistered [ 1769.246874] Key type ._llcrypt unregistered [ 1774.584320] Key type ._llcrypt registered [ 1774.585964] Key type .llcrypt registered [ 1774.796519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1774.819991] alg: No test for adler32 (adler32-zlib) [ 1775.732255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.824768] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1775.826921] LNet: *** cfs_fail_loc=e003, val=100*** [ 1776.519484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1776.542456] LNet: *** cfs_fail_loc=e003, val=100*** [ 1776.544329] LNet: Skipped 2 previous similar messages [ 1776.559483] Ping reply received with mapping: [ 1776.561390] found metadata reply NI[192.168.203.49@tcp] [ 1776.564795] 31 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1776.568297] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1776.570464] 5a 5a 5a 5a 5a 5a 5a 5a [ 1776.711516] LNet: 99765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1776.717527] LNetError: 99765:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.729855] LNet: Removed LNI 192.168.203.49@tcp [ 1777.228486] Key type .llcrypt unregistered [ 1777.233116] Key type ._llcrypt unregistered [ 1786.126089] Key type ._llcrypt registered [ 1786.128101] Key type .llcrypt registered [ 1787.255806] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 17:54:34 (1763333674) [ 1787.846078] Key type .llcrypt unregistered [ 1787.847777] Key type ._llcrypt unregistered [ 1793.116463] Key type ._llcrypt registered [ 1793.118091] Key type .llcrypt registered [ 1793.256450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1793.267345] alg: No test for adler32 (adler32-zlib) [ 1794.253143] LNetError: 100801:0:(api-ni.c:2725:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1794.580351] Key type .llcrypt unregistered [ 1794.582109] Key type ._llcrypt unregistered [ 1798.676304] Key type ._llcrypt registered [ 1798.678276] Key type .llcrypt registered [ 1799.728484] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 17:54:47 (1763333687) [ 1800.583719] Key type .llcrypt unregistered [ 1800.585042] Key type ._llcrypt unregistered [ 1806.702934] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1806.705463] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1809.519839] Key type ._llcrypt registered [ 1809.521213] Key type .llcrypt registered [ 1809.743766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1809.757538] alg: No test for adler32 (adler32-zlib) [ 1810.721131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1810.771647] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1822.851441] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 17:55:09 (1763333709) [ 1822.955923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1822.988093] LNet: 102569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.994624] LNetError: 102569:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1823.033477] LNet: Removed LNI 10.1.2.3@tcp [ 1823.074077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.763539] Key type .llcrypt unregistered [ 1823.764996] Key type ._llcrypt unregistered [ 1828.804222] Key type ._llcrypt registered [ 1828.805371] Key type .llcrypt registered [ 1829.004553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1829.020417] alg: No test for adler32 (adler32-zlib) [ 1830.008055] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1830.078109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1830.205752] LNet: 102859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.217215] LNetError: 102859:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.271750] LNet: Removed LNI 192.168.203.49@tcp [ 1831.883535] Key type .llcrypt unregistered [ 1831.886544] Key type ._llcrypt unregistered [ 1841.177453] Key type ._llcrypt registered [ 1841.178716] Key type .llcrypt registered [ 1842.358466] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 17:55:29 (1763333729) [ 1843.091314] Key type .llcrypt unregistered [ 1843.094445] Key type ._llcrypt unregistered [ 1848.077344] Key type ._llcrypt registered [ 1848.079021] Key type .llcrypt registered [ 1848.318086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1848.329639] alg: No test for adler32 (adler32-zlib) [ 1854.602128] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 17:55:42 (1763333742) [ 1854.839458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1854.875859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1854.940507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1854.975831] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1855.618485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1855.640441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1855.670272] LNet: Removed LNI 192.168.203.49@tcp [ 1855.939539] LNet: 104595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1856.237085] Key type .llcrypt unregistered [ 1856.238752] Key type ._llcrypt unregistered [ 1866.465734] Key type ._llcrypt registered [ 1866.467427] Key type .llcrypt registered [ 1867.957848] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 17:55:54 (1763333754) [ 1868.334213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1868.347035] alg: No test for adler32 (adler32-zlib) [ 1869.253893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1869.339124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1869.367870] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1870.166139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1870.207481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.49@tcp [ 1870.230185] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1870.252616] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1870.260345] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1870, 0, 0 [ 1870.268180] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1870.277823] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1870.287453] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1870.292745] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1870, 0, 0 [ 1870.300809] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1870.307408] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1870.319170] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1870.326675] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1870, 0, 1920 [ 1870.334207] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1870.337246] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1870.343110] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1870.348251] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1870, 0, 1920 [ 1870.357169] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1876.653607] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 17:56:03 (1763333763) [ 1877.235107] LNet: 106133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.257090] LNetError: 106133:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.364736] LNet: Removed LNI 192.168.203.49@tcp [ 1877.961697] Key type .llcrypt unregistered [ 1877.963386] Key type ._llcrypt unregistered [ 1884.070071] Key type ._llcrypt registered [ 1884.082033] Key type .llcrypt registered [ 1884.439469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1884.455521] alg: No test for adler32 (adler32-zlib) [ 1885.407244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1885.429251] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1892.903245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.149@tcp [ 1900.845824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.932507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1901.001209] LNet: Added LNI 192.168.203.49@tcp1 [8/256/0/180] [ 1901.961548] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1902.048398] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1902.050931] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1902.056844] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1904.383329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1904.395677] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1914.383195] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 17:56:41 (1763333801) [ 1914.568175] LNet: 107631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.580746] LNetError: 107631:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.585451] LNet: Removed LNI 192.168.203.49@tcp [ 1915.615278] LNet: Removed LNI 192.168.203.49@tcp1 [ 1915.621361] LNetError: 107631:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1915.636883] LNetError: 107631:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1915.665598] LNet: Removed LNI 10.1.2.3@tcp99 [ 1916.439536] Key type .llcrypt unregistered [ 1916.441257] Key type ._llcrypt unregistered [ 1921.337229] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1921.348362] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1923.809978] Key type ._llcrypt registered [ 1923.811764] Key type .llcrypt registered [ 1923.956769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1923.970413] alg: No test for adler32 (adler32-zlib) [ 1924.869592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1924.918073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1924.958860] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1925.628758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1925.639112] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1927.478907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1927.531180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1927.570985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1927.588941] LNetError: 108370:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1927.616349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1927.702394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 1928.065124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1928.136752] LNet: 108390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.147294] LNetError: 108390:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1929.183239] LNet: Removed LNI 192.168.203.49@tcp [ 1929.188709] LNetError: 108390:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1929.206969] LNet: Removed LNI 10.1.2.3@tcp [ 1929.788643] Key type .llcrypt unregistered [ 1929.789929] Key type ._llcrypt unregistered [ 1939.120631] Key type ._llcrypt registered [ 1939.122389] Key type .llcrypt registered [ 1940.347753] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 17:57:07 (1763333827) [ 1940.860514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1940.877462] alg: No test for adler32 (adler32-zlib) [ 1941.807601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.864500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1941.922103] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1942.687343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.49@tcp [ 1942.791625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg349-client@tcp [ 1948.810909] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 17:57:15 (1763333835) [ 1948.920707] LNet: 109909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.934619] LNetError: 109909:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.947361] LNet: Removed LNI 192.168.203.49@tcp [ 1949.694768] Key type .llcrypt unregistered [ 1949.700046] Key type ._llcrypt unregistered [ 1954.578983] Key type ._llcrypt registered [ 1954.585569] Key type .llcrypt registered [ 1954.774400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1954.789165] alg: No test for adler32 (adler32-zlib) [ 1955.705471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1955.756599] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1955.794822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1961.657817] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 17:57:29 (1763333849) [ 1962.923505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1964.849953] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 17:57:31 (1763333851) [ 1966.274835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1967.772444] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 17:57:35 (1763333855) [ 1968.075705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1968.101954] LNet: 111335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.107903] LNetError: 111335:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1968.118680] LNet: Removed LNI 192.168.203.49@tcp [ 1968.136218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1968.181610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1968.218362] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1968.885457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1968.919553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1975.790444] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 17:57:42 (1763333862) [ 1975.906640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1975.922672] LNet: 112038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1975.935924] LNetError: 112038:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.946607] LNet: Removed LNI 192.168.203.49@tcp [ 1975.973206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1976.013980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1976.042730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1976.644995] Key type .llcrypt unregistered [ 1976.646764] Key type ._llcrypt unregistered [ 1988.608057] Key type ._llcrypt registered [ 1988.609789] Key type .llcrypt registered [ 1990.500316] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 17:57:57 (1763333877) [ 1991.098451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1991.137865] alg: No test for adler32 (adler32-zlib) [ 1992.362659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.476366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1992.519239] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 1993.742513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1993.805681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2002.167155] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 17:58:09 (1763333889) [ 2002.412486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2002.460514] LNet: 113595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2002.471466] LNetError: 113595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2002.485925] LNet: Removed LNI 192.168.203.49@tcp [ 2002.529728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.596994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2002.635508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2010.854233] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 17:58:17 (1763333897) [ 2010.971769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2010.999912] LNet: 114169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.033046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2011.155679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2011.778277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2011.804024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2011.866777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2012.265569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2012.351544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2012.504548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2012.522237] LNet: 114208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2012.551072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2012.601902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2012.921972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2012.951592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2013.017212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2013.309415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2013.334197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2020.291634] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 17:58:27 (1763333907) [ 2020.544342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2020.583077] LNet: 114806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2020.614681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2020.762576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2020.825614] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2021.714905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2021.740550] LNet: Added LNI 192.168.203.49@tcp2 [8/256/0/180] [ 2022.742538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2022.805454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2023.137256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2023.186507] LNetError: 115088:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.204171] LNet: Removed LNI 192.168.203.49@tcp [ 2023.219821] LNet: Removed LNI 192.168.203.49@tcp2 [ 2023.270643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2023.363374] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2023.374246] LNet: Added LNI 192.168.203.49@tcp2 [8/256/0/180] [ 2023.377649] LNet: 115094:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2023.791858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2023.972623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2025.055298] LNet: Removed LNI 192.168.203.49@tcp [ 2025.057368] LNetError: 115131:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2025.071169] LNetError: 115131:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2025.099791] LNet: Removed LNI 192.168.203.49@tcp2 [ 2025.126832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2025.172186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2025.203354] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2026.072860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2026.102214] LNet: Added LNI 192.168.203.49@tcp2 [8/256/0/180] [ 2027.320779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2027.356798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2027.434443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2027.677849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2027.701750] LNet: 115416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2027.708244] LNet: 115416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2027.714624] LNetError: 115416:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2027.734727] LNet: Removed LNI 192.168.203.49@tcp [ 2028.777528] LNet: Removed LNI 192.168.203.49@tcp2 [ 2028.808638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2028.898283] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2028.918185] LNet: Added LNI 192.168.203.49@tcp2 [8/256/0/180] [ 2028.928221] LNet: 115422:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2029.551772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2029.693056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2029.706303] LNet: Removed LNI 192.168.203.49@tcp [ 2030.766243] LNet: Removed LNI 192.168.203.49@tcp2 [ 2030.812780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2030.932130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2030.979476] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2031.867150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2031.886557] LNet: Added LNI 192.168.203.49@tcp2 [8/256/0/180] [ 2032.916229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2032.992733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2033.241297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2033.261589] LNetError: 115738:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.267155] LNetError: 115738:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2033.273843] LNet: Removed LNI 192.168.203.49@tcp [ 2033.291517] LNet: Removed LNI 192.168.203.49@tcp2 [ 2033.316393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.383637] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2033.389239] LNet: Added LNI 192.168.203.49@tcp2 [8/256/0/180] [ 2033.395880] LNet: 115744:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2033.906677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2034.116646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2035.167283] LNet: Removed LNI 192.168.203.49@tcp [ 2035.182229] LNet: Removed LNI 192.168.203.49@tcp2 [ 2035.210280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2035.259683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2035.286753] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2036.017441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2036.047716] LNet: Added LNI 192.168.203.49@tcp2 [8/256/0/180] [ 2036.980925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2037.032304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2037.133623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2037.431422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2037.476852] LNet: 116066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.479995] LNet: 116066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2038.559610] LNet: Removed LNI 192.168.203.49@tcp [ 2039.592286] LNet: Removed LNI 192.168.203.49@tcp2 [ 2039.611980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2039.690945] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2039.702805] LNet: Added LNI 192.168.203.49@tcp2 [8/256/0/180] [ 2039.705914] LNet: 116072:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2040.270954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2040.515926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2040.551676] LNet: Removed LNI 192.168.203.49@tcp [ 2041.574744] LNet: Removed LNI 192.168.203.49@tcp2 [ 2041.604217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2041.681425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2041.727530] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2042.589784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2042.610353] LNet: Added LNI 192.168.203.49@tcp2 [8/256/0/180] [ 2043.472592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2043.496422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2043.569380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2043.933610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2043.957211] LNetError: 116406:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2043.962444] LNetError: 116406:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2043.967992] LNet: Removed LNI 192.168.203.49@tcp [ 2043.979991] LNet: Removed LNI 192.168.203.49@tcp2 [ 2043.998109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2044.050067] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2044.057851] LNet: Added LNI 192.168.203.49@tcp2 [8/256/0/180] [ 2044.064923] LNet: 116412:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2044.566433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2044.925759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2045.983223] LNet: Removed LNI 192.168.203.49@tcp [ 2046.010689] LNet: Removed LNI 192.168.203.49@tcp2 [ 2046.040324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2046.092804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2046.119142] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2046.840954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2046.873087] LNet: Added LNI 192.168.203.49@tcp2 [8/256/0/180] [ 2047.936174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2048.005413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2048.070089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2048.168653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2048.611868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2049.695171] LNet: Removed LNI 192.168.203.49@tcp [ 2050.730454] LNet: Removed LNI 192.168.203.49@tcp2 [ 2050.764801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2050.857566] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2050.862225] LNet: Added LNI 192.168.203.49@tcp2 [8/256/0/180] [ 2051.411482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2059.494628] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2061.104737] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 17:59:08 (1763333948) [ 2061.272205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2061.298044] LNet: 117530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.303429] LNet: 117530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2061.324132] LNetError: 117530:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2061.326929] LNetError: 117530:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2061.335782] LNet: Removed LNI 192.168.203.49@tcp [ 2062.374787] LNet: Removed LNI 192.168.203.49@tcp2 [ 2062.397214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2062.684640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2062.708306] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2063.414733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2063.437651] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2064.225586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2064.248739] LNet: Removed LNI 192.168.203.49@tcp [ 2064.258790] LNet: Removed LNI 10.1.2.3@tcp [ 2064.280559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2064.328644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2064.357317] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2064.363830] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2064.648851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2064.672263] LNet: Removed LNI 192.168.203.49@tcp [ 2064.690144] LNet: Removed LNI 10.1.2.3@tcp [ 2070.922260] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 17:59:18 (1763333958) [ 2071.051711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2071.140205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2071.284736] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2071.421816] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2071.424946] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2071.428623] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2073.717443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2073.744655] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2074.573368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2074.613991] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2076.963407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.49@tcp [ 2077.002396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2077.119856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.49@tcp -m PUT -e local_timeout -r 1 [ 2077.262697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2077.333091] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2077.335319] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2077.608764] LNetError: 118462:0:(lib-move.c:4947:lnet_parse()) 192.168.203.49@tcp, src 192.168.203.49@tcp: Dropping PUT (error -108 looking up sender) [ 2079.647214] LNet: Removed LNI 192.168.203.49@tcp [ 2079.662741] LNet: Removed LNI 10.1.2.3@tcp [ 2080.317593] Key type .llcrypt unregistered [ 2080.322682] Key type ._llcrypt unregistered [ 2092.393710] Key type ._llcrypt registered [ 2092.395307] Key type .llcrypt registered [ 2093.806910] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 17:59:40 (1763333980) [ 2094.289341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2094.312689] alg: No test for adler32 (adler32-zlib) [ 2095.296418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2095.714145] LNet: 119771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2096.423140] Key type .llcrypt unregistered [ 2096.425909] Key type ._llcrypt unregistered [ 2101.528111] Key type ._llcrypt registered [ 2101.533756] Key type .llcrypt registered [ 2101.783928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2101.811420] alg: No test for adler32 (adler32-zlib) [ 2102.771741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2102.809487] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2109.659447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.149@tcp [ 2109.946252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2110.450333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.149@tcp [ 2110.602182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2110.620163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2110.640472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2110.666591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2110.695363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2110.723680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2110.745884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2110.780539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2110.794788] LNet: 120216:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2110.809432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2110.837301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2110.862748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2110.898619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2110.918974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2110.948896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2111.428714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2111.631384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2111.669272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2111.712574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2111.735967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2111.791083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2111.827580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2111.865638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2111.890867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2111.913190] LNet: 120333:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2111.927477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2111.950952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2111.976556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2112.009732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2112.037149] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.149@tcp [ 2112.043477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2112.079899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2112.714508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.149@tcp [ 2113.047529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2113.092354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2113.129540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2113.188490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2113.230938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2113.277047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2113.323883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2113.369843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2113.409329] LNet: 120450:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2113.429299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2113.465535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2113.500474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2113.569632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2113.634829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2113.692095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2114.260783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2114.472530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2114.503716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2114.532346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2114.550560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2114.589831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2114.615536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2114.635609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2114.664730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2114.703545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2114.732541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2114.752986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2114.779909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2114.815722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2114.849908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2115.347209] LNet: 120612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.354562] LNetError: 120612:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2115.370787] LNet: Removed LNI 192.168.203.49@tcp [ 2115.770372] Key type .llcrypt unregistered [ 2115.782361] Key type ._llcrypt unregistered [ 2130.805675] Key type ._llcrypt registered [ 2130.807453] Key type .llcrypt registered [ 2131.804297] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 18:00:19 (1763334019) [ 2132.252773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2132.268323] alg: No test for adler32 (adler32-zlib) [ 2133.182609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.596979] LNet: 121497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2134.180393] Key type .llcrypt unregistered [ 2134.185577] Key type ._llcrypt unregistered [ 2139.038078] Key type ._llcrypt registered [ 2139.039665] Key type .llcrypt registered [ 2139.224258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2139.240441] alg: No test for adler32 (adler32-zlib) [ 2140.130534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2140.156214] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2146.965231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.149@tcp [ 2154.306850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.347799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2154.377850] LNet: Added LNI 192.168.203.49@tcp1 [8/256/0/180] [ 2155.274472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2155.750440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.149@tcp [ 2155.982956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2156.005661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2156.035328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2156.064710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2156.097243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2156.121624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2156.153336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2156.181706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2156.191316] LNet: 122145:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2156.209399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2156.230045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2156.258343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2156.287778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2156.316240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2156.338480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2156.763602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2156.920335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2156.951206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2156.977539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2157.008411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2157.038739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2157.067314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2157.091404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2157.124880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2157.143157] LNet: 122262:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2157.154780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2157.179537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2157.205679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2157.232593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2157.267508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2157.294803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2157.755367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.149@tcp [ 2157.944639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2157.971659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2158.008863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2158.034202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2158.061388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2158.083261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2158.108695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2158.144374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2158.160530] LNet: 122379:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2158.181932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2158.219559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2158.253811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2158.278942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2158.298875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2158.326411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2158.763343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2158.951181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2158.983662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2159.004261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2159.030572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2159.069896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2159.098221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2159.120792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2159.159712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2159.192864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2159.218651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2159.257033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2159.281447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2159.304294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2159.333914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2161.629707] LNet: 122576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.636136] LNetError: 122576:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2161.641350] LNet: Removed LNI 192.168.203.49@tcp [ 2161.654573] LNet: Removed LNI 192.168.203.49@tcp1 [ 2161.960728] Key type .llcrypt unregistered [ 2161.963084] Key type ._llcrypt unregistered [ 2175.520739] Key type ._llcrypt registered [ 2175.522962] Key type .llcrypt registered [ 2176.820027] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1941 sec ========== 18:01:04 (1763334064) [ 2177.312948] Key type .llcrypt unregistered [ 2177.314568] Key type ._llcrypt unregistered [ 2186.732444] Key type ._llcrypt registered [ 2186.734125] Key type .llcrypt registered [ 2187.108823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2187.128812] alg: No test for adler32 (adler32-zlib) [ 2188.183565] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 2188.557255] LNet: Added LNI 192.168.203.49@tcp [8/256/0/180] [ 2190.327265] Key type lgssc registered [ 2191.876820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2241.417223] Lustre: Mounted lustre-client [ 2245.836945] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2257.774826] LustreError: 125841:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff88a2850a6800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2257.797077] LustreError: 125841:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2257.831847] Lustre: Unmounted lustre-client [ 2275.301524] Key type lgssc unregistered [ 2275.583987] LNet: 126322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2275.592703] LNetError: 126322:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2275.612311] LNet: Removed LNI 192.168.203.49@tcp [ 2276.341163] Key type .llcrypt unregistered [ 2276.345253] Key type ._llcrypt unregistered