[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 442959021 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002335] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006685] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010152] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013088] *** VALIDATE selinux *** [ 0.021561] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026196] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028049] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031068] *** VALIDATE tmpfs *** [ 0.032397] *** VALIDATE proc *** [ 0.034028] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041017] Spectre V2 : User space: Vulnerable [ 0.041656] Speculative Store Bypass: Vulnerable [ 0.043909] debug: unmapping init [mem 0xffffffff96e59000-0xffffffff96e60fff] [ 0.045873] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046730] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048021] ... bit width: 48 [ 0.049015] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054332] rcu: Hierarchical SRCU implementation. [ 0.056599] smp: Bringing up secondary CPUs ... [ 0.057687] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.065026] smp: Brought up 1 node, 4 CPUs [ 0.067019] smpboot: Max logical packages: 1 [ 0.068015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141812] node 0 deferred pages initialised in 72ms [ 0.146014] devtmpfs: initialized [ 0.147262] x86/mm: Memory block size: 128MB [ 0.149973] gcov: version magic: 0x41383552 [ 0.151294] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153225] pinctrl core: initialized pinctrl subsystem [ 0.154156] [ 0.154673] ************************************************************* [ 0.155015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156016] ** ** [ 0.157015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158015] ** ** [ 0.159022] ** This means that this kernel is built to expose internal ** [ 0.160014] ** IOMMU data structures, which may compromise security on ** [ 0.161016] ** your system. ** [ 0.162017] ** ** [ 0.163015] ** If you see this message and you are not debugging the ** [ 0.164022] ** kernel, report this immediately to your vendor! ** [ 0.165030] ** ** [ 0.166014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167013] ************************************************************* [ 0.168695] NET: Registered protocol family 16 [ 0.169488] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173056] cpuidle: using governor menu [ 0.174907] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178607] PCI: Using configuration type 1 for base access [ 0.180144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189152] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.192020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.195191] cryptd: max_cpu_qlen set to 1000 [ 0.198248] ACPI: Added _OSI(Module Device) [ 0.199007] ACPI: Added _OSI(Processor Device) [ 0.201013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.207073] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.211396] ACPI: Interpreter enabled [ 0.212049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214011] ACPI: Using IOAPIC for interrupt routing [ 0.215113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.219433] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.229553] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.232051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.234021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.237068] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.241310] acpiphp: Slot [2] registered [ 0.242079] acpiphp: Slot [5] registered [ 0.243105] acpiphp: Slot [6] registered [ 0.244110] acpiphp: Slot [3] registered [ 0.245000] acpiphp: Slot [4] registered [ 0.245000] acpiphp: Slot [7] registered [ 0.246060] acpiphp: Slot [8] registered [ 0.247055] acpiphp: Slot [9] registered [ 0.248050] acpiphp: Slot [10] registered [ 0.250067] acpiphp: Slot [11] registered [ 0.251064] acpiphp: Slot [12] registered [ 0.252081] acpiphp: Slot [13] registered [ 0.253083] acpiphp: Slot [14] registered [ 0.254071] acpiphp: Slot [15] registered [ 0.256096] acpiphp: Slot [16] registered [ 0.257161] acpiphp: Slot [17] registered [ 0.258070] acpiphp: Slot [18] registered [ 0.259065] acpiphp: Slot [19] registered [ 0.260054] acpiphp: Slot [20] registered [ 0.260931] acpiphp: Slot [21] registered [ 0.262050] acpiphp: Slot [22] registered [ 0.262946] acpiphp: Slot [23] registered [ 0.263081] acpiphp: Slot [24] registered [ 0.264080] acpiphp: Slot [25] registered [ 0.265084] acpiphp: Slot [26] registered [ 0.266026] acpiphp: Slot [27] registered [ 0.266901] acpiphp: Slot [28] registered [ 0.268050] acpiphp: Slot [29] registered [ 0.268921] acpiphp: Slot [30] registered [ 0.269086] acpiphp: Slot [31] registered [ 0.269931] PCI host bridge to bus 0000:00 [ 0.271012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.272011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.273013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.274015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.277042] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.279027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.281193] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.283780] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.285931] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.290756] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.294387] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.297022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.299020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.303021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.305622] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.308952] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.312052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.314848] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.319013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.329026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.334019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.339850] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.349019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.354015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.370021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.378332] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.385017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.388903] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.401017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.409148] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.411315] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.413243] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.415324] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.416159] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.421095] iommu: Default domain type: Passthrough [ 0.422461] SCSI subsystem initialized [ 0.423150] ACPI: bus type USB registered [ 0.424222] usbcore: registered new interface driver usbfs [ 0.425171] usbcore: registered new interface driver hub [ 0.426063] usbcore: registered new device driver usb [ 0.427127] pps_core: LinuxPPS API ver. 1 registered [ 0.429010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.431052] PTP clock support registered [ 0.433092] EDAC MC: Ver: 3.0.0 [ 0.435421] PCI: Using ACPI for IRQ routing [ 0.436571] NetLabel: Initializing [ 0.438006] NetLabel: domain hash size = 128 [ 0.438882] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.440063] NetLabel: unlabeled traffic allowed by default [ 0.441237] vgaarb: loaded [ 0.442107] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.442980] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.449853] clocksource: Switched to clocksource kvm-clock [ 0.550095] VFS: Disk quotas dquot_6.6.0 [ 0.551659] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.553837] *** VALIDATE ramfs *** [ 0.555127] *** VALIDATE hugetlbfs *** [ 0.556533] pnp: PnP ACPI init [ 0.558781] pnp: PnP ACPI: found 6 devices [ 0.574342] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.577554] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.579120] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.581219] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.583711] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.585957] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.588482] NET: Registered protocol family 2 [ 0.591565] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.597297] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.600843] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.606523] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.610265] TCP: Hash tables configured (established 65536 bind 65536) [ 0.612360] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.615069] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.617541] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.620135] NET: Registered protocol family 1 [ 0.622931] RPC: Registered named UNIX socket transport module. [ 0.625420] RPC: Registered udp transport module. [ 0.627379] RPC: Registered tcp transport module. [ 0.629188] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.631465] NET: Registered protocol family 44 [ 0.633080] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.635100] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.637098] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.639633] PCI: CLS 0 bytes, default 64 [ 0.641275] Unpacking initramfs... [ 1.994764] debug: unmapping init [mem 0xffff9e307cc64000-0xffff9e307ffcffff] [ 1.997555] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.999012] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.001062] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.451979] Initialise system trusted keyrings [ 2.453325] Key type blacklist registered [ 2.454524] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.460771] zbud: loaded [ 2.463360] *** VALIDATE nfs *** [ 2.464152] *** VALIDATE nfs4 *** [ 2.465295] pstore: using deflate compression [ 2.469108] Platform Keyring initialized [ 2.547764] NET: Registered protocol family 38 [ 2.550425] Key type asymmetric registered [ 2.552620] Asymmetric key parser 'x509' registered [ 2.554655] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.557511] io scheduler mq-deadline registered [ 2.559250] io scheduler kyber registered [ 2.561248] io scheduler bfq registered [ 2.563080] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.565958] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.568984] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.571753] ACPI: Power Button [PWRF] [ 2.576565] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.582526] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.597185] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.625596] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.654887] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.659644] Non-volatile memory driver v1.3 [ 2.661369] Linux agpgart interface v0.103 [ 2.690638] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 2.692966] vda: detected capacity change from 0 to 76677120 [ 2.707349] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.709594] vdb: detected capacity change from 0 to 1073741824 [ 2.717631] libphy: Fixed MDIO Bus: probed [ 2.724350] usbcore: registered new interface driver usbserial_generic [ 2.726933] usbserial: USB Serial support registered for generic [ 2.729422] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.733458] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.735209] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.737324] mousedev: PS/2 mouse device common for all mice [ 2.740333] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.741304] rtc_cmos 00:05: RTC can wake from S4 [ 2.745853] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.746619] rtc_cmos 00:05: registered as rtc0 [ 2.749985] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.750032] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.751501] intel_pstate: CPU model not supported [ 2.755244] hid: raw HID events driver (C) Jiri Kosina [ 2.757119] usbcore: registered new interface driver usbhid [ 2.758536] usbhid: USB HID core driver [ 2.759575] drop_monitor: Initializing network drop monitor service [ 2.761057] Initializing XFRM netlink socket [ 2.762417] NET: Registered protocol family 10 [ 2.764227] Segment Routing with IPv6 [ 2.765102] NET: Registered protocol family 17 [ 2.766202] mpls_gso: MPLS GSO support [ 2.769737] RAS: Correctable Errors collector initialized. [ 2.771224] AVX version of gcm_enc/dec engaged. [ 2.772130] AES CTR mode by8 optimization enabled [ 2.833139] sched_clock: Marking stable (2833121241, 0)->(3739918341, -906797100) [ 2.835396] registered taskstats version 1 [ 2.836517] Loading compiled-in X.509 certificates [ 2.837919] zswap: loaded using pool lzo/zbud [ 2.860684] Key type big_key registered [ 2.869759] Key type encrypted registered [ 2.870644] ima: No TPM chip found, activating TPM-bypass! [ 2.871815] ima: Allocated hash algorithm: sha1 [ 2.872775] ima: No architecture policies found [ 2.873919] evm: Initialising EVM extended attributes: [ 2.874926] evm: security.selinux [ 2.875600] evm: security.ima [ 2.876246] evm: security.capability [ 2.876948] evm: HMAC attrs: 0x1 [ 2.878718] rtc_cmos 00:05: setting system clock to 2026-09-05 12:44:31 UTC (1788612271) [ 2.883324] debug: unmapping init [mem 0xffffffff97e03000-0xffffffff97ffffff] [ 2.885200] debug: unmapping init [mem 0xffffffff96b82000-0xffffffff96e58fff] [ 2.894252] Write protecting the kernel read-only data: 28672k [ 2.896274] debug: unmapping init [mem 0xffffffff95203000-0xffffffff953fffff] [ 2.898958] debug: unmapping init [mem 0xffffffff95b14000-0xffffffff95bfffff] [ 2.925667] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.933717] systemd[1]: Detected virtualization kvm. [ 2.935495] systemd[1]: Detected architecture x86-64. [ 2.937224] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.958706] systemd[1]: No hostname configured. [ 2.960507] systemd[1]: Set hostname to . [ 2.962680] random: systemd: uninitialized urandom read (16 bytes read) [ 2.964140] systemd[1]: Initializing machine ID from random generator. [ 3.062130] random: systemd: uninitialized urandom read (16 bytes read) [ 3.063919] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.066716] random: systemd: uninitialized urandom read (16 bytes read) [ 3.068104] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.070824] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.540423] device-mapper: uevent: version 1.0.3 [ 3.542148] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ 3.746418] random: fast init done [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.135706] virtio_net virtio0 ens2: renamed from eth0 [ 4.162199] scsi host0: ata_piix [ 4.196918] scsi host1: ata_piix [ 4.198489] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.200598] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.383517] random: crng init done [ 8.384456] random: 7 urandom warning(s) missed due to ratelimiting [ 8.491844] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.041088] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.325674] printk: systemd: 24 output lines suppressed due to ratelimiting [ 10.564202] SELinux: Disabled at runtime. [ 10.621535] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.628779] systemd[1]: Detected virtualization kvm. [ 10.630227] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.176571] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.180335] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.185664] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.190797] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.194736] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.202755] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.225174] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ 11.433580] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 11.771791] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.165562] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.219466] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.355942] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.370912] EDAC sbridge: Ver: 1.1.2 [ 13.463785] Key type dns_resolver registered [ 13.766205] NFS: Registering the id_resolver key type [ 13.769051] Key type id_resolver registered [ 13.771099] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg231-client login: [ 49.842259] hrtimer: interrupt took 1773440 ns [ 58.504989] libcfs: loading out-of-tree module taints kernel. [ 58.801119] Key type ._llcrypt registered [ 58.814922] Key type .llcrypt registered [ 59.669912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 59.698564] alg: No test for adler32 (adler32-zlib) [ 61.458513] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 62.753674] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 64.536258] Key type lgssc registered [ 67.169304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 212.222654] Lustre: Mounted lustre-client - version 2.17.58_2_ge9089f3 [ 217.916642] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 232.389621] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing check_logdir /tmp/testlogs/ [ 238.053276] Lustre: lustre-OST0000-osc-ffff9e30d4315000: disconnect after 23s idle [ 238.193436] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing yml_node [ 242.730802] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 245.431341] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 248.636548] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 250.720468] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Sep 5 08:48:37 EDT 2026 [ 268.082974] Lustre: Unmounted lustre-client [ 293.422854] Key type lgssc unregistered [ 293.759517] LNet: 8416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 293.773467] LNetError: 8416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 293.817310] LNet: Removed LNI 192.168.202.31@tcp [ 294.832249] Key type .llcrypt unregistered [ 294.839492] Key type ._llcrypt unregistered [ 324.502613] Key type ._llcrypt registered [ 324.505235] Key type .llcrypt registered [ 324.725588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 324.769674] alg: No test for adler32 (adler32-zlib) [ 325.720648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 325.802288] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 325.827915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 330.132611] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 330.358194] LNet: 9296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.373294] LNetError: 9296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 330.418570] LNet: Removed LNI 192.168.202.31@tcp [ 331.321469] Key type .llcrypt unregistered [ 331.327155] Key type ._llcrypt unregistered [ 338.590234] Key type ._llcrypt registered [ 338.608267] Key type .llcrypt registered [ 340.112625] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 08:50:06 (1788612606) [ 340.798144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 340.822665] alg: No test for adler32 (adler32-zlib) [ 341.753678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.819780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.835611] LNet: 9706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.850707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.890393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 341.903338] LNet: 9713:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 349.013777] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 08:50:16 (1788612616) [ 349.113557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.140531] LNet: 10286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.163252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.267072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 356.195516] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 08:50:23 (1788612623) [ 356.371527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.444141] LNet: 10874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.518580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.681616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 363.982562] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 08:50:31 (1788612631) [ 364.228994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.259363] LNet: 11465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.288526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.376566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 370.981966] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 08:50:38 (1788612638) [ 371.180980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.229321] LNet: 12045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.257565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.321650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 371.754773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 379.093964] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 08:50:46 (1788612646) [ 379.244800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.264318] LNet: 12634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.319836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.422943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 379.782076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 380.341397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 389.927224] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 08:50:56 (1788612656) [ 390.258706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.297547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.384851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 397.565662] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 08:51:04 (1788612664) [ 397.707720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.740710] LNet: 13821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.763351] LNet: 13821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 397.813538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.871225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 397.910651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 398.418738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 398.472516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 398.966244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 398.995633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 399.622704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 399.685493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 400.040932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 400.073853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 400.671794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 400.716995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 401.132829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 401.215922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 401.799137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 401.844979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 402.415065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 402.442560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 403.129586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 403.184137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 403.704868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 403.774710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 404.224330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 404.303866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 404.812186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 404.896982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 411.509906] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 08:51:18 (1788612678) [ 411.758777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.869407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.026736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 412.066231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 419.439354] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 08:51:26 (1788612686) [ 419.744261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.814851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.901713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 419.989174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 426.305815] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 08:51:33 (1788612693) [ 426.507656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.540419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.623747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 426.676069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 432.587489] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 08:51:39 (1788612699) [ 432.676868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.721689] LNet: 16259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.727364] LNet: 16259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 432.751619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.891390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 432.934673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 439.318969] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 08:51:46 (1788612706) [ 439.575814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.612617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.673803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 439.711517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 446.035307] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 08:51:53 (1788612713) [ 446.231409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.261740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.317212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 446.339974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 452.881865] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 08:51:59 (1788612719) [ 452.973824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.027597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.162185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 453.637810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.594649] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 08:52:07 (1788612727) [ 460.829288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.871591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.025500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.237284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 467.367915] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 08:52:14 (1788612734) [ 467.494876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.542384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.685854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 467.874652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 473.969442] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 08:52:21 (1788612741) [ 474.056648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.102503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.427319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 474.663478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 479.459215] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 08:52:27 (1788612747) [ 479.691529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.735048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.876280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 479.914715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 491.180867] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 08:52:36 (1788612756) [ 491.579385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.643528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.768130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 491.806951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 500.995773] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 08:52:48 (1788612768) [ 501.679917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.724910] LNet: 21706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 501.727218] LNet: 21706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 501.812290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.041039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 509.720833] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 08:52:56 (1788612776) [ 509.906532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.978984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.118113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 515.781956] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 08:53:03 (1788612783) [ 515.837349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.868139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.937394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 516.283455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 521.797222] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 08:53:09 (1788612789) [ 522.080953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.117769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.198475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 522.225784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 527.658491] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 08:53:15 (1788612795) [ 527.864844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.910712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.965874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 527.985606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 533.673945] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 08:53:21 (1788612801) [ 533.765721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.810847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.934659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 533.977606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 541.188838] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 08:53:27 (1788612807) [ 541.338776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.396872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.467413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 541.509843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 547.474668] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 08:53:35 (1788612815) [ 547.629260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.689299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.769158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 547.821741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 547.840842] LNetError: 25823:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 547.870892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 547.905993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 553.247130] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 08:53:40 (1788612820) [ 553.603284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.663717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.786801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.v8khzS9H6q [ 553.901366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.v8khzS9H6q [ 553.995836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.v8khzS9H6q [ 561.480315] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 08:53:48 (1788612828) [ 561.640367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.721760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.856739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 561.974857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 562.053311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 567.822585] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 08:53:55 (1788612835) [ 568.305099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.342925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.400676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 568.416272] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 570.493964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 572.213966] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 574.192578] Key type lgssc registered [ 575.418175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 578.569587] LNetError: 27584:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.202.31@tcp99 has no available nets [ 580.868142] Key type lgssc unregistered [ 581.169824] LNetError: 28689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 582.188134] LNet: Removed LNI 192.168.202.31@tcp [ 582.727381] Key type .llcrypt unregistered [ 582.728758] Key type ._llcrypt unregistered [ 588.546539] Key type ._llcrypt registered [ 588.548327] Key type .llcrypt registered [ 589.736994] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 08:54:17 (1788612857) [ 590.551388] Key type .llcrypt unregistered [ 590.561080] Key type ._llcrypt unregistered [ 596.168560] Key type ._llcrypt registered [ 596.170513] Key type .llcrypt registered [ 596.309067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 596.323423] alg: No test for adler32 (adler32-zlib) [ 597.198852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.286228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 597.302541] LNet: 29598:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 597.309564] LNet: 29598:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 597.319245] LNet: 29598:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 597.339697] LNet: Message forwarding will be enabled in 2s [ 599.395597] LNet: Message forwarding enabled [ 599.771080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 599.797518] LNet: Message forwarding will stop in 2s to 6s [ 603.492412] LNet: Message forwarding disabled [ 604.296308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 604.346065] LNet: 29687:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 604.355226] LNet: 29687:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 604.363190] LNet: 29687:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 604.380706] LNet: Message forwarding will be enabled in 2s [ 604.537839] LNet: 29700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 605.290541] Key type .llcrypt unregistered [ 605.294847] Key type ._llcrypt unregistered [ 618.712684] Key type ._llcrypt registered [ 618.714266] Key type .llcrypt registered [ 620.389541] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 08:54:47 (1788612887) [ 620.751346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 620.786340] alg: No test for adler32 (adler32-zlib) [ 621.725161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.785141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 627.530058] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 08:54:54 (1788612894) [ 627.664787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.686458] LNet: 31093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.702701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.747494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 627.793106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 627.826143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 627.847427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 627.879845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 627.910450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 627.934709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 627.962030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 627.997834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 628.026398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 628.047653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 628.065231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 628.102387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 628.131372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 628.150731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 628.181478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 628.208565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 628.235471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 628.277403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 628.298350] LNetError: 31142:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 633.441582] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 08:55:00 (1788612900) [ 633.545927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.580538] LNet: 31708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.681850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.782681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 639.321430] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 08:55:06 (1788612906) [ 640.985922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg231-server [ 642.204870] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 08:55:09 (1788612909) [ 642.367986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.385221] LNet: 32644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.395805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.431848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 642.448646] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 643.325354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[32-36]@tcp [ 643.349842] LNet: 32777:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 643.430973] LNetError: 32648:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.32@tcp is being used as a gateway but routing feature is not turned on [ 643.443256] LNetError: 32648:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.32@tcp has gone from up to down [ 651.356788] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 08:55:18 (1788612918) [ 653.068886] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg231-server [ 655.297892] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 08:55:22 (1788612922) [ 655.454584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.506856] LNet: 33708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.520973] LNetError: 33708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.551367] LNet: Removed LNI 192.168.202.31@tcp [ 655.623728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.723029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 655.757716] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 656.730818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[32-37/2]@tcp [ 656.774737] LNet: 33830:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 656.784245] LNet: 33830:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 656.815510] LNetError: 33724:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.36@tcp has gone from up to down [ 656.841997] LNetError: 33724:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 656.846060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[32-37/2]@tcp [ 656.858839] LNetError: 33717:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.32@tcp is being used as a gateway but routing feature is not turned on [ 656.858852] LNetError: 33717:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 665.016284] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 08:55:32 (1788612932) [ 665.272904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.306363] LNet: 34424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.314157] LNetError: 34424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.332373] LNet: Removed LNI 192.168.202.31@tcp [ 665.389417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.497962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 665.596434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.663456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.729449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 665.782013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.820534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.872853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 665.955415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.004416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.086107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 666.179177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.216056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.268477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 666.376501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.411524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.501934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 666.595797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.647814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.703701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 666.772463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.804803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.893777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 666.968365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 667.007551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.090685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 667.172771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 667.192472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.311556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 667.401658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 667.431519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.478359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 667.566437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 667.609901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.702811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 675.050675] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 08:55:42 (1788612942) [ 676.521193] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg231-server [ 678.365040] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 08:55:45 (1788612945) [ 681.181415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg231-server [ 683.102726] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 08:55:50 (1788612950) [ 683.153084] LNet: 35818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.156656] LNet: 35818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 684.024941] Key type .llcrypt unregistered [ 684.026851] Key type ._llcrypt unregistered [ 690.716183] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 690.844844] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 690.857044] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 690.867355] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 693.492155] Key type ._llcrypt registered [ 693.493901] Key type .llcrypt registered [ 693.690821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 693.723176] alg: No test for adler32 (adler32-zlib) [ 694.659432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.770540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 694.807092] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 695.464992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 695.485708] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 697.299671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 697.320576] LNet: Removed LNI 10.1.2.3@tcp [ 697.484956] LNet: 36545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.493750] LNetError: 36545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 697.517854] LNet: Removed LNI 192.168.202.31@tcp [ 698.127349] Key type .llcrypt unregistered [ 698.129456] Key type ._llcrypt unregistered [ 709.784317] Key type ._llcrypt registered [ 709.788895] Key type .llcrypt registered [ 711.168403] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 08:56:18 (1788612978) [ 711.688394] Key type .llcrypt unregistered [ 711.690866] Key type ._llcrypt unregistered [ 717.631463] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 717.862050] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 717.887795] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 717.913220] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 720.635405] Key type ._llcrypt registered [ 720.642452] Key type .llcrypt registered [ 720.874232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 720.883884] alg: No test for adler32 (adler32-zlib) [ 721.816877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.862895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 721.881338] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 722.514364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 722.535610] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 725.434704] LNet: 38034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.440657] LNetError: 38034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 725.447977] LNet: Removed LNI 10.1.2.3@tcp [ 725.458824] LNet: Removed LNI 192.168.202.31@tcp [ 726.098131] Key type .llcrypt unregistered [ 726.100272] Key type ._llcrypt unregistered [ 739.510595] Key type ._llcrypt registered [ 739.513414] Key type .llcrypt registered [ 741.025873] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 08:56:47 (1788613007) [ 741.572244] Key type .llcrypt unregistered [ 741.573771] Key type ._llcrypt unregistered [ 748.977219] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 749.144783] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 749.151283] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 749.165203] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 751.798281] Key type ._llcrypt registered [ 751.800573] Key type .llcrypt registered [ 752.031697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 752.058274] alg: No test for adler32 (adler32-zlib) [ 753.091969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.147286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 753.166762] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 755.345725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 755.373645] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 757.695996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 757.709670] LNet: Removed LNI 10.1.2.3@tcp [ 757.729628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 757.773421] LNet: Removed LNI 10.1.2.31@tcp [ 757.785631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 757.817898] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 758.667792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 758.702467] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 759.533874] LNet: 39942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.541813] LNetError: 39942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 759.548126] LNet: Removed LNI 10.1.2.31@tcp [ 759.554640] LNet: Removed LNI 10.1.2.3@tcp [ 760.182449] Key type .llcrypt unregistered [ 760.184098] Key type ._llcrypt unregistered [ 772.924755] Key type ._llcrypt registered [ 772.928928] Key type .llcrypt registered [ 774.338995] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 08:57:21 (1788613041) [ 774.739295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 774.769499] alg: No test for adler32 (adler32-zlib) [ 775.762392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.820988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 775.841545] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 776.600643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 776.619808] LNet: 40891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.624561] LNetError: 40891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.651514] LNet: Removed LNI 192.168.202.31@tcp [ 776.668304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.706948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.31@tcp [ 776.731151] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 777.472093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.31@tcp [ 777.514493] LNet: Removed LNI 192.168.202.31@tcp [ 777.601850] LNet: 41026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.220896] Key type .llcrypt unregistered [ 778.226588] Key type ._llcrypt unregistered [ 790.258770] Key type ._llcrypt registered [ 790.263716] Key type .llcrypt registered [ 791.538081] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 08:57:38 (1788613058) [ 793.197704] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg231-server [ 794.873614] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 08:57:42 (1788613062) [ 795.446177] Key type .llcrypt unregistered [ 795.449101] Key type ._llcrypt unregistered [ 801.569191] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 801.769966] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 801.775182] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 801.778717] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 804.423956] Key type ._llcrypt registered [ 804.430236] Key type .llcrypt registered [ 804.695230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 804.709731] alg: No test for adler32 (adler32-zlib) [ 805.614937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.664264] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 806.403558] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 808.301461] LNet: 42858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.308677] LNetError: 42858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.314189] LNet: Removed LNI 192.168.202.31@tcp1 [ 808.323672] LNet: Removed LNI 10.1.2.3@tcp2 [ 808.758459] Key type .llcrypt unregistered [ 808.759989] Key type ._llcrypt unregistered [ 817.988144] Key type ._llcrypt registered [ 817.992231] Key type .llcrypt registered [ 819.369215] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 08:58:06 (1788613086) [ 819.888703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 819.899737] alg: No test for adler32 (adler32-zlib) [ 820.823528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.868260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 820.900280] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 821.725328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.745298] LNet: 43824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.754962] LNetError: 43824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.794339] LNet: Removed LNI 192.168.202.31@tcp [ 821.829777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.908476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 821.941890] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 827.317203] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 08:58:14 (1788613094) [ 827.439668] LNet: 44413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.444451] LNetError: 44413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.477329] LNet: Removed LNI 192.168.202.31@tcp [ 827.953546] Key type .llcrypt unregistered [ 827.955195] Key type ._llcrypt unregistered [ 832.324060] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 832.486537] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 832.491806] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 832.506124] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 834.965625] Key type ._llcrypt registered [ 834.968913] Key type .llcrypt registered [ 835.141066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 835.151538] alg: No test for adler32 (adler32-zlib) [ 836.108940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.161124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 836.179858] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 836.724167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 836.740805] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 838.417587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 838.433870] LNet: 45154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.440591] LNetError: 45154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 838.444791] LNet: Removed LNI 10.1.2.3@tcp [ 838.454126] LNet: Removed LNI 192.168.202.31@tcp [ 838.469862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.523659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 838.542884] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 838.546090] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 838.912827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 838.953793] LNet: 45181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.958632] LNetError: 45181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 838.962887] LNetError: 45181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 838.968874] LNet: Removed LNI 10.1.2.3@tcp [ 838.993730] LNet: Removed LNI 192.168.202.31@tcp [ 839.008859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.071790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 839.090554] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 839.093897] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 839.519337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 839.535823] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 840.604290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.620740] LNet: 45343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.624695] LNetError: 45343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 840.630425] LNetError: 45343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 840.635213] LNet: Removed LNI 10.1.2.3@tcp [ 840.639913] LNet: Removed LNI 192.168.202.31@tcp [ 840.663409] LNet: Removed LNI 192.168.202.31@tcp2 [ 840.686081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.754530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 840.807405] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 840.814316] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 840.825149] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 848.001702] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 08:58:35 (1788613115) [ 848.142041] LNet: 45932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.169933] LNetError: 45932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 848.189492] LNetError: 45932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 848.200127] LNet: Removed LNI 10.1.2.3@tcp [ 848.201717] LNet: Removed LNI 192.168.202.31@tcp [ 848.236850] LNet: Removed LNI 192.168.202.31@tcp2 [ 849.136698] Key type .llcrypt unregistered [ 849.140700] Key type ._llcrypt unregistered [ 856.592694] Key type ._llcrypt registered [ 856.595910] Key type .llcrypt registered [ 857.078355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 857.138234] alg: No test for adler32 (adler32-zlib) [ 858.082456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.151740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 858.187656] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 859.026408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.041625] LNet: 46332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.046274] LNetError: 46332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.054805] LNet: Removed LNI 192.168.202.31@tcp [ 859.069592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.134378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 859.157911] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 859.742957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.765750] LNet: 46362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.771865] LNetError: 46362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.786896] LNet: Removed LNI 192.168.202.31@tcp [ 859.824680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.894762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 859.919288] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 860.581427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.630504] LNet: Removed LNI 192.168.202.31@tcp [ 860.645629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.716957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 860.748197] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 861.381471] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 861.556990] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 861.566043] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 861.572288] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 863.982774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.000915] LNet: 46470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.006182] LNet: 46470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 864.016855] LNetError: 46470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.022839] LNetError: 46470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 864.059084] LNet: Removed LNI 192.168.202.31@tcp [ 864.092036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.178806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 864.207091] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 871.813902] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 08:58:59 (1788613139) [ 872.172662] LNet: 47067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.189660] LNetError: 47067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.218335] LNet: Removed LNI 192.168.202.31@tcp [ 873.141234] Key type .llcrypt unregistered [ 873.143806] Key type ._llcrypt unregistered [ 879.156978] Key type ._llcrypt registered [ 879.158970] Key type .llcrypt registered [ 879.323318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 879.338882] alg: No test for adler32 (adler32-zlib) [ 880.246738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.303127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 880.351825] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 881.041684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.059864] LNet: 47479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.065375] LNetError: 47479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.078724] LNet: Removed LNI 192.168.202.31@tcp [ 881.091221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.141641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 881.161345] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 888.472107] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 08:59:15 (1788613155) [ 888.581353] LNet: 48064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.614324] LNetError: 48064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.644512] LNet: Removed LNI 192.168.202.31@tcp [ 889.324578] Key type .llcrypt unregistered [ 889.327421] Key type ._llcrypt unregistered [ 894.942726] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 895.410915] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 895.415319] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 895.422923] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 898.259664] Key type ._llcrypt registered [ 898.264956] Key type .llcrypt registered [ 898.537075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 898.586425] alg: No test for adler32 (adler32-zlib) [ 899.694717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.815834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 899.861902] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 901.117739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.150769] LNet: 48518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.158327] LNetError: 48518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.178505] LNet: Removed LNI 192.168.202.31@tcp [ 901.196343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.265811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 901.290253] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 907.972681] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 08:59:35 (1788613175) [ 908.391449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.406226] LNet: 49118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.415776] LNetError: 49118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.432372] LNet: Removed LNI 192.168.202.31@tcp [ 908.447287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.519802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 915.146364] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 08:59:42 (1788613182) [ 915.255163] LNet: 49692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.834795] Key type .llcrypt unregistered [ 915.836434] Key type ._llcrypt unregistered [ 921.490169] Key type ._llcrypt registered [ 921.491321] Key type .llcrypt registered [ 921.643630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 921.656072] alg: No test for adler32 (adler32-zlib) [ 922.539302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.583976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 922.619335] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 923.310174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.327799] LNet: 50095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.333174] LNetError: 50095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.343732] LNet: Removed LNI 192.168.202.31@tcp [ 923.360744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.410808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 923.428899] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 929.053421] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 08:59:56 (1788613196) [ 929.157363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.202795] LNet: 50681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.217343] LNetError: 50681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.244581] LNet: Removed LNI 192.168.202.31@tcp [ 929.266378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.387617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 935.603892] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:00:02 (1788613202) [ 937.130749] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 938.668348] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:00:06 (1788613206) [ 940.051967] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 941.729647] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:00:09 (1788613209) [ 943.103776] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 944.550895] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:00:11 (1788613211) [ 945.876348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 947.759263] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:00:14 (1788613214) [ 949.227545] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 950.733889] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 09:00:18 (1788613218) [ 950.849200] LNet: 52734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.440565] Key type .llcrypt unregistered [ 951.443549] Key type ._llcrypt unregistered [ 957.540491] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 957.696924] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 957.704502] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 957.713629] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 960.299519] Key type ._llcrypt registered [ 960.305924] Key type .llcrypt registered [ 960.581569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.595206] alg: No test for adler32 (adler32-zlib) [ 961.589440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.710545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 961.760987] LNet: Added LNI 192.168.202.31@tcp [8/512/0/180] [ 961.767259] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 961.769094] LNet: 53057:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 961.878897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.909714] LNet: 53076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.927772] LNetError: 53076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 961.941839] LNet: Removed LNI 10.1.2.3@tcp [ 961.943756] LNetError: 53076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.965858] LNet: Removed LNI 192.168.202.31@tcp [ 962.012840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.111695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 962.143708] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 962.283278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.297872] LNet: 53100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.302608] LNetError: 53100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.316680] LNet: Removed LNI 192.168.202.31@tcp [ 962.337989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.418722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 962.446418] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 962.449284] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 962.451044] LNet: 53108:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 962.540310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.581397] LNet: 53126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.585550] LNetError: 53126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 962.592645] LNet: Removed LNI 10.1.2.3@tcp [ 962.606445] LNetError: 53126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.635817] LNet: Removed LNI 192.168.202.31@tcp [ 962.670643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.763813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 962.793230] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 962.804855] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 962.807210] LNet: 53134:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 963.095512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.110050] LNet: 53160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.113160] LNetError: 53160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 963.116042] LNet: Removed LNI 10.1.2.3@tcp [ 963.117065] LNetError: 53160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.126901] LNet: Removed LNI 192.168.202.31@tcp [ 963.146673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.184596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 963.198772] LNet: Added LNI 192.168.202.31@tcp [8/256/0/182] [ 963.202534] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 963.247919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.257136] LNet: 53188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.259802] LNetError: 53188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 963.262634] LNet: Removed LNI 10.1.2.3@tcp [ 963.264103] LNetError: 53188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.270417] LNet: Removed LNI 192.168.202.31@tcp [ 963.281651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.309328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 963.318612] LNet: Added LNI 192.168.202.31@tcp [8/256/0/182] [ 963.320641] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 963.321686] LNet: 53196:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 963.353798] LNet: Removed LNI 10.1.2.3@tcp [ 963.364735] LNet: Removed LNI 192.168.202.31@tcp [ 963.750475] Key type .llcrypt unregistered [ 963.754369] Key type ._llcrypt unregistered [ 974.938954] Key type ._llcrypt registered [ 974.944240] Key type .llcrypt registered [ 976.509773] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 09:00:43 (1788613243) [ 978.503311] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 980.214977] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:00:47 (1788613247) [ 980.828494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 980.851499] alg: No test for adler32 (adler32-zlib) [ 981.833921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.905209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 981.963224] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 983.011888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.31 [ 983.054771] LNet: 54458:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 983.066750] LNet: Removed LNI 192.168.202.31@tcp [ 990.749478] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:00:57 (1788613257) [ 990.966378] LNet: 55021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.984637] LNetError: 55021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.009714] LNet: Removed LNI 192.168.202.31@tcp [ 991.637550] Key type .llcrypt unregistered [ 991.640549] Key type ._llcrypt unregistered [ 998.612376] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 998.903616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1000.624489] Key type ._llcrypt registered [ 1000.628573] Key type .llcrypt registered [ 1000.775461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1000.789770] alg: No test for adler32 (adler32-zlib) [ 1002.094041] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1008.695904] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:01:15 (1788613275) [ 1008.897651] LNet: 55944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.902118] LNetError: 55944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1008.911891] LNet: Removed LNI 10.1.2.3@tcp [ 1009.437543] Key type .llcrypt unregistered [ 1009.438693] Key type ._llcrypt unregistered [ 1015.156101] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1015.361965] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1015.377226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1015.389470] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1017.002798] Key type ._llcrypt registered [ 1017.009989] Key type .llcrypt registered [ 1017.180314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1017.197906] alg: No test for adler32 (adler32-zlib) [ 1018.181280] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1023.998781] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:01:31 (1788613291) [ 1025.491306] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1027.097065] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:01:34 (1788613294) [ 1027.354664] LNet: 57116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.362642] LNetError: 57116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1027.399177] LNet: Removed LNI 10.1.2.3@tcp [ 1027.930709] Key type .llcrypt unregistered [ 1027.932372] Key type ._llcrypt unregistered [ 1034.319778] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1034.550132] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1034.561089] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1034.568680] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1037.277844] Key type ._llcrypt registered [ 1037.287911] Key type .llcrypt registered [ 1037.532688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1037.558415] alg: No test for adler32 (adler32-zlib) [ 1038.521773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1038.586120] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1045.884456] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:01:53 (1788613313) [ 1046.237878] LNet: 58343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.243927] LNetError: 58343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1046.260762] LNet: Removed LNI 10.1.2.3@tcp [ 1046.843564] Key type .llcrypt unregistered [ 1046.845324] Key type ._llcrypt unregistered [ 1053.101855] Key type ._llcrypt registered [ 1053.103575] Key type .llcrypt registered [ 1053.280919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1053.314994] alg: No test for adler32 (adler32-zlib) [ 1054.244923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1054.266266] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1061.690820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1062.278955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_interrupt -r 1 [ 1062.321109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_interrupt -r 1 [ 1062.366578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1062.400427] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1062.500628] LNet: 1 local NIs in recovery (showing 1): 192.168.202.31@tcp [ 1062.665634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1062.727587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1062.771487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1062.802877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1062.826436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1062.873907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1062.935511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1062.974519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1063.000237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1063.036452] LNet: 58779:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1063.059361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1063.097788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1063.149505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1063.176382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1063.200754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1063.224953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1063.273829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1063.826902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_dropped -r 1 [ 1063.864214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_dropped -r 1 [ 1063.888535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1063.902415] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1064.084011] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1064.114872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1064.155868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1064.212490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1064.262441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1064.291739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1064.321785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1064.358228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1064.393707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1064.409216] LNet: 58890:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1064.427990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1064.452963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1064.482678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1064.509651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1064.543602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1064.572667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1064.604304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1065.104994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_aborted -r 1 [ 1065.131486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_aborted -r 1 [ 1065.166669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1065.189744] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1065.194901] LNet: Skipped 1 previous similar message [ 1065.408608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1065.432242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1065.458996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1065.489849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1065.532526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1065.561895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1065.587696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1065.619555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1065.650889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1065.668666] LNet: 59001:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1065.688789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1065.734929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1065.785592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1065.825834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1065.879350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1065.930722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1065.966542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1066.571446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_no_route -r 1 [ 1066.600969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_no_route -r 1 [ 1066.641923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1066.871934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1066.932835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1066.999389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1067.065968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1067.129192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1067.187583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1067.227905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1067.283754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1067.324900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1067.359303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1067.390308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1067.412267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1067.455797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1067.493947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1067.525451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1067.557194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1068.030515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_timeout -r 1 [ 1068.063763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_timeout -r 1 [ 1068.094597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1068.113486] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1068.117941] LNet: Skipped 1 previous similar message [ 1068.345841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1068.375713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1068.415421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1068.451478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1068.495374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1068.526087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1068.554498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1068.580390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1068.612724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1068.638508] LNet: 59224:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1068.652408] LNet: 59224:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1068.678869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1068.715481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1068.750095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1068.792117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1068.830172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1068.874340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1068.891727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1069.545407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_error -r 1 [ 1069.580878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_error -r 1 [ 1069.615491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1069.834725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1069.867728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1069.898262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1069.926293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1069.955543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1069.983671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1070.018222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1070.045755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1070.078908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1070.116987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1070.154158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1070.195372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1070.237512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1070.279766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1070.306976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1070.338205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1070.634976] LNet: 59376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1070.642717] LNetError: 59376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.660324] LNet: Removed LNI 192.168.202.31@tcp [ 1072.110528] Key type .llcrypt unregistered [ 1072.113523] Key type ._llcrypt unregistered [ 1089.175056] Key type ._llcrypt registered [ 1089.181338] Key type .llcrypt registered [ 1090.969232] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:02:37 (1788613357) [ 1092.113568] Key type .llcrypt unregistered [ 1092.116258] Key type ._llcrypt unregistered [ 1099.941728] Key type ._llcrypt registered [ 1099.944623] Key type .llcrypt registered [ 1100.148366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1100.166731] alg: No test for adler32 (adler32-zlib) [ 1101.231533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1101.266062] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1110.277478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1121.910098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1121.969667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1121.992749] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1123.162940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_interrupt -r 1 [ 1123.188323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_interrupt -r 1 [ 1123.213705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_interrupt -r 1 [ 1123.245777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_interrupt -r 1 [ 1123.279140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_interrupt -r 1 [ 1123.310187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_interrupt -r 1 [ 1123.350649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_interrupt -r 1 [ 1123.376596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_interrupt -r 1 [ 1123.407511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1123.431663] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1123.437577] LNet: 1 local NIs in recovery (showing 1): 192.168.202.31@tcp [ 1123.667163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1123.694485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1123.730209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1123.764660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1123.816404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1123.844407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1123.870208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1123.902496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1123.922661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1123.933850] LNet: 60841:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1123.947585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1123.965822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1123.988579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.019483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.046203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.073547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.104299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1124.569301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_dropped -r 1 [ 1124.594898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_dropped -r 1 [ 1124.630926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_dropped -r 1 [ 1124.669270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_dropped -r 1 [ 1124.707540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_dropped -r 1 [ 1124.741275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_dropped -r 1 [ 1124.775442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_dropped -r 1 [ 1124.802888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_dropped -r 1 [ 1124.836069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1124.865699] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1124.870449] LNet: Skipped 2 previous similar messages [ 1125.096267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1125.125359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1125.154548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1125.184124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1125.208492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1125.239551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1125.273588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1125.301423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1125.331252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1125.343491] LNet: 60964:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1125.356100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1125.383524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1125.413276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1125.435691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1125.462892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1125.489785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1125.524782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1125.953688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_aborted -r 1 [ 1125.994404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_aborted -r 1 [ 1126.035048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_aborted -r 1 [ 1126.073261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_aborted -r 1 [ 1126.102613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_aborted -r 1 [ 1126.131217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_aborted -r 1 [ 1126.161245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_aborted -r 1 [ 1126.203907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_aborted -r 1 [ 1126.255512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1126.273782] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1126.276135] LNet: Skipped 2 previous similar messages [ 1126.493991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.516448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1126.554052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.590196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.613726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1126.637510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.660525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.692239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.734889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1126.755230] LNet: 61087:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1126.776417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1126.803240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1126.831510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.862890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.908400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.940782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.968215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1127.346157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_no_route -r 1 [ 1127.372617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_no_route -r 1 [ 1127.400947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_no_route -r 1 [ 1127.429717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_no_route -r 1 [ 1127.457547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_no_route -r 1 [ 1127.504573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_no_route -r 1 [ 1127.550978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_no_route -r 1 [ 1127.580771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_no_route -r 1 [ 1127.612769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1127.825872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1127.854811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1127.881217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1127.903810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1127.928882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1127.960338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1127.985866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1128.010344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1128.038995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1128.068801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1128.122380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1128.157289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1128.183497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1128.208825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1128.230747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1128.253739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1128.773328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_timeout -r 1 [ 1128.821213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_timeout -r 1 [ 1128.855885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_timeout -r 1 [ 1128.894456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_timeout -r 1 [ 1128.931897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_timeout -r 1 [ 1128.990408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_timeout -r 1 [ 1129.039196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_timeout -r 1 [ 1129.081218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_timeout -r 1 [ 1129.101091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1129.112154] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1129.116106] LNet: Skipped 5 previous similar messages [ 1129.365819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1129.394758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1129.420330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1129.447919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1129.469668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1129.500770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1129.534772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1129.569443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1129.598811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1129.617926] LNet: 61334:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1129.622883] LNet: 61334:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1129.639846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1129.675204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1129.712919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1129.741857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1129.768836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.791827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.826696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1130.305769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_error -r 1 [ 1130.332664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_error -r 1 [ 1130.374845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_error -r 1 [ 1130.407944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_error -r 1 [ 1130.439536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_error -r 1 [ 1130.464661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_error -r 1 [ 1130.496447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_error -r 1 [ 1130.526791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_error -r 1 [ 1130.604613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1130.790236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1130.821900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1130.862950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1130.885232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1130.904732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1130.974955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1131.028987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1131.106824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1131.151380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1131.202670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1131.232850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1131.277427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1131.304938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1131.331915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1131.356259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1131.380306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1136.086648] LNet: 61532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.091800] LNetError: 61532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.096441] LNet: Removed LNI 192.168.202.31@tcp [ 1137.145734] LNet: Removed LNI 192.168.202.31@tcp1 [ 1137.604783] Key type .llcrypt unregistered [ 1137.617534] Key type ._llcrypt unregistered [ 1154.638313] Key type ._llcrypt registered [ 1154.640090] Key type .llcrypt registered [ 1156.140952] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:03:43 (1788613423) [ 1156.912451] Key type .llcrypt unregistered [ 1156.915630] Key type ._llcrypt unregistered [ 1162.616916] Key type ._llcrypt registered [ 1162.618711] Key type .llcrypt registered [ 1162.843611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1162.875266] alg: No test for adler32 (adler32-zlib) [ 1163.797196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1163.829662] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1171.798851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1172.492620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e remote_dropped -r 1 [ 1172.539944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e remote_dropped -r 1 [ 1172.635779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1172.668751] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1172.900670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.943533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1172.981214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.019260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.073709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1173.112033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.154568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.183560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.213139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.230315] LNet: 62778:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.253885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.287786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.321748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.363155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.404702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.443712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.488533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1174.059150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e remote_error -r 1 [ 1174.092605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e remote_error -r 1 [ 1174.115857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1174.132093] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1174.342491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.369294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.403909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.434054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.482334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1174.521713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.564375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.611680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.647898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.669551] LNet: 62889:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.690646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.722341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.748270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.775756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.800421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.823602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.840847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1175.354796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e remote_timeout -r 1 [ 1175.381815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e remote_timeout -r 1 [ 1175.425623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1175.457573] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1175.680117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.715046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1175.745747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.783465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.823635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1175.873581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.894305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.920540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.942458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.954237] LNet: 63000:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.965383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.985626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.016300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.043560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.070913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.094658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.127159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1176.335516] LNet: 63041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1176.344243] LNetError: 63041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1176.363253] LNet: Removed LNI 192.168.202.31@tcp [ 1176.761460] Key type .llcrypt unregistered [ 1176.764465] Key type ._llcrypt unregistered [ 1192.679319] Key type ._llcrypt registered [ 1192.681377] Key type .llcrypt registered [ 1193.888532] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:04:21 (1788613461) [ 1194.576486] Key type .llcrypt unregistered [ 1194.577941] Key type ._llcrypt unregistered [ 1199.994920] Key type ._llcrypt registered [ 1199.997721] Key type .llcrypt registered [ 1200.152478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1200.173892] alg: No test for adler32 (adler32-zlib) [ 1201.119071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1201.160328] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1208.270177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1220.876991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1220.928173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1220.993021] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1222.705697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e remote_dropped -r 1 [ 1222.754491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e remote_dropped -r 1 [ 1222.817157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e remote_dropped -r 1 [ 1222.874324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e remote_dropped -r 1 [ 1222.941589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e remote_dropped -r 1 [ 1223.052658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e remote_dropped -r 1 [ 1223.100753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e remote_dropped -r 1 [ 1223.130434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e remote_dropped -r 1 [ 1223.168685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1223.187260] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1223.193452] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.131@tcp [ 1223.403293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1223.462990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1223.498942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1223.541838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1223.635575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1223.669577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1223.721407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1223.753162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1223.778600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1223.791709] LNet: 64506:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1223.801547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1223.835608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1223.854746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1223.878708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1223.914576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1223.947622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1223.979185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1224.023091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1224.670113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e remote_error -r 1 [ 1224.697371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e remote_error -r 1 [ 1224.743819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e remote_error -r 1 [ 1224.776985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e remote_error -r 1 [ 1224.807539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e remote_error -r 1 [ 1224.847796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e remote_error -r 1 [ 1224.876880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e remote_error -r 1 [ 1224.923543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e remote_error -r 1 [ 1224.966881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1224.979696] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1224.983702] LNet: Skipped 2 previous similar messages [ 1225.225323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1225.254419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1225.290613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1225.327538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1225.355927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1225.384070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1225.408633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1225.426935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1225.451823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1225.462799] LNet: 64631:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1225.474803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1225.501695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1225.530867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1225.567909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1225.600788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1225.638540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1225.683458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1225.723969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.508603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e remote_timeout -r 1 [ 1226.546946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e remote_timeout -r 1 [ 1226.643661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e remote_timeout -r 1 [ 1226.725273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e remote_timeout -r 1 [ 1226.779618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e remote_timeout -r 1 [ 1226.841820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e remote_timeout -r 1 [ 1226.881756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e remote_timeout -r 1 [ 1226.920739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e remote_timeout -r 1 [ 1226.965329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1226.990913] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1226.995975] LNet: Skipped 1 previous similar message [ 1227.321816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1227.377937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1227.428411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1227.474323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1227.511490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1227.536602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1227.590916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1227.661800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1227.755889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1227.834452] LNet: 64756:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1227.892086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1227.988928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1228.057992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.102140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.173223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.279464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1228.335889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1228.413945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.230794] LNet: 64834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1233.253410] LNetError: 64834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1233.266881] LNet: Removed LNI 192.168.202.31@tcp [ 1233.283692] LNet: Removed LNI 192.168.202.31@tcp1 [ 1233.907578] Key type .llcrypt unregistered [ 1233.910475] Key type ._llcrypt unregistered [ 1251.993843] Key type ._llcrypt registered [ 1251.995516] Key type .llcrypt registered [ 1253.233132] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:05:20 (1788613520) [ 1255.220971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1257.092719] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:05:24 (1788613524) [ 1258.253514] Key type .llcrypt unregistered [ 1258.256609] Key type ._llcrypt unregistered [ 1265.101896] Key type ._llcrypt registered [ 1265.106794] Key type .llcrypt registered [ 1265.255105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1265.273333] alg: No test for adler32 (adler32-zlib) [ 1266.191516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1266.213977] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1273.825402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1274.480921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e network_timeout -r 1 [ 1274.510472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e network_timeout -r 1 [ 1274.538491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1274.554195] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1274.773752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1274.791440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1274.809357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1274.830806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1274.852625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1274.878424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1274.899750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1274.925607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1274.948756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1274.977334] LNet: 66375:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1274.995331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1275.029537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1275.056695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1275.098848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1275.131136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1275.163175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1275.427706] LNet: 66414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1275.430958] LNetError: 66414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1275.442442] LNet: Removed LNI 192.168.202.31@tcp [ 1275.821430] Key type .llcrypt unregistered [ 1275.824401] Key type ._llcrypt unregistered [ 1294.067680] Key type ._llcrypt registered [ 1294.070556] Key type .llcrypt registered [ 1294.228617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1294.244276] alg: No test for adler32 (adler32-zlib) [ 1295.185791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1295.211505] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1302.835274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1312.695555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1312.771286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1312.809652] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1314.433310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e network_timeout -r 1 [ 1314.488459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e network_timeout -r 1 [ 1314.526971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e network_timeout -r 1 [ 1314.571442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e network_timeout -r 1 [ 1314.604573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e network_timeout -r 1 [ 1314.636598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e network_timeout -r 1 [ 1314.681382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e network_timeout -r 1 [ 1314.726366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e network_timeout -r 1 [ 1314.767566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1314.786814] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1315.028118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1315.062551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1315.096971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1315.129239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1315.159324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1315.187661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1315.232989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1315.259716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1315.282453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1315.297667] LNet: 67290:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1315.319441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1315.345929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1315.372910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1315.406716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1315.439863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1315.465320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1319.724451] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1319.776176] LNet: 1 local NIs in recovery (showing 1): 192.168.202.31@tcp [ 1320.018688] LNet: 67364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1320.028228] LNetError: 67364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1320.039613] LNet: Removed LNI 192.168.202.31@tcp [ 1320.054870] LNet: Removed LNI 192.168.202.31@tcp1 [ 1320.419696] Key type .llcrypt unregistered [ 1320.422063] Key type ._llcrypt unregistered [ 1337.398326] Key type ._llcrypt registered [ 1337.400737] Key type .llcrypt registered [ 1339.830824] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:06:45 (1788613605) [ 1340.311989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1340.336687] alg: No test for adler32 (adler32-zlib) [ 1341.485385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1341.569971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1341.597757] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1342.780760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1342.829103] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1344.257612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 1344.381495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1344.450168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1344.499439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1344.599421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1344.699101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1344.773685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.31@tcp [ 1344.841341] LNetError: 68461:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.31@tcp to recovery [ 1345.632899] LNet: 1 local NIs in recovery (showing 1): 192.168.202.31@tcp [ 1345.635245] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1347.680594] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1348.529457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1351.776788] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1352.243974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1352.291730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1352.388627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1352.412544] LNet: 68568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1352.426186] LNetError: 68568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1352.432191] LNet: Removed LNI 192.168.202.31@tcp [ 1352.442945] LNet: Removed LNI 192.168.202.31@tcp1 [ 1352.482236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1352.597213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1352.642166] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1353.575961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1353.594618] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1354.598269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 1354.670161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1354.745327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1354.805335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1354.870918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1354.936587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1354.993280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1355.032655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.31@tcp [ 1355.046832] LNetError: 68840:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.31@tcp to recovery [ 1355.616812] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1358.627891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1361.760872] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1361.774499] LNet: Skipped 1 previous similar message [ 1366.759417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1366.797488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1366.836424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1373.794932] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:07:21 (1788613641) [ 1373.975147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1374.031396] LNet: 69553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1374.042214] LNetError: 69553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.046396] LNetError: 69553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1374.052620] LNet: Removed LNI 192.168.202.31@tcp [ 1374.082084] LNet: Removed LNI 192.168.202.31@tcp1 [ 1374.129266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.222092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1374.266939] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1375.199167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1375.228819] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1376.207414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 1376.264373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1376.300261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1376.330972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1376.382348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1376.408566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1376.433121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.31@tcp --health 0 [ 1376.543466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1377.248507] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.31@tcp [ 1377.250346] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1377.288617] LNet: Skipped 1 previous similar message [ 1392.020590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1392.187661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1392.221726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1392.259304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.31@tcp --health 500 [ 1392.436634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1399.329687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1399.413425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1399.430236] LNet: 70049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1399.436512] LNetError: 70049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1399.442467] LNetError: 70049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1399.446204] LNet: Removed LNI 192.168.202.31@tcp [ 1399.459969] LNet: Removed LNI 192.168.202.31@tcp1 [ 1399.490566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1399.579591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1399.621280] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1400.600399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1400.616495] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1401.396391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 1401.434463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1401.509819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1401.562508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1401.638357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1401.705868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1401.784268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1401.887897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.31@tcp --health 0 [ 1402.656411] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1402.661070] LNet: Skipped 3 previous similar messages [ 1403.282046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1413.887291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1413.932925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1413.994206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1422.744875] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:08:09 (1788613689) [ 1423.595883] LNet: 71096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1423.614572] LNetError: 71096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1423.626877] LNetError: 71096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1423.641511] LNet: Removed LNI 192.168.202.31@tcp [ 1423.653367] LNet: Removed LNI 192.168.202.31@tcp1 [ 1424.264682] Key type .llcrypt unregistered [ 1424.273240] Key type ._llcrypt unregistered [ 1431.954548] Key type ._llcrypt registered [ 1431.956780] Key type .llcrypt registered [ 1432.181240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1432.204579] alg: No test for adler32 (adler32-zlib) [ 1433.126119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1433.159242] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1442.722414] Lustre: DEBUG MARKER: Initial discovery [ 1442.937126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.131@tcp [ 1445.393745] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1445.445920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1445.488332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.131@tcp [ 1445.550689] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1445.571984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1445.637668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.131@tcp -r 1 -m GET -l 3 [ 1445.709527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.131@tcp -r 1 -m GET -e local_error [ 1445.741884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.131@tcp -r 1 -m PUT -l 6 [ 1446.496469] LNet: 1 local NIs in recovery (showing 1): 192.168.202.31@tcp [ 1447.040592] Lustre: DEBUG MARKER: Force oleg231-client.virtnet to discover 192.168.202.131@tcp (in background) [ 1447.119463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.131@tcp [ 1448.619541] Lustre: DEBUG MARKER: Force oleg231-server to discover 192.168.202.31@tcp [ 1450.215512] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1451.037989] Lustre: DEBUG MARKER: Wait for 71849 [ 1452.404217] Lustre: DEBUG MARKER: Finished wait on 71849 [ 1452.575800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1452.626608] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1452.636558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1452.863485] LNet: 72304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.867814] LNetError: 72304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.886192] LNet: Removed LNI 192.168.202.31@tcp [ 1453.303787] Key type .llcrypt unregistered [ 1453.305579] Key type ._llcrypt unregistered [ 1468.844145] Key type ._llcrypt registered [ 1468.848194] Key type .llcrypt registered [ 1470.171519] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:08:57 (1788613737) [ 1470.595272] Key type .llcrypt unregistered [ 1470.597264] Key type ._llcrypt unregistered [ 1477.171408] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1477.432857] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1477.438520] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1477.440928] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1479.961211] Key type ._llcrypt registered [ 1479.962851] Key type .llcrypt registered [ 1480.134216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1480.148875] alg: No test for adler32 (adler32-zlib) [ 1481.148432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.239531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1481.288812] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1481.930814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1481.949877] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1490.678425] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:09:17 (1788613757) [ 1490.836234] LNet: 74368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1490.856883] LNetError: 74368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1490.862050] LNet: Removed LNI 10.1.2.3@tcp [ 1490.903451] LNet: Removed LNI 192.168.202.31@tcp [ 1491.680420] Key type .llcrypt unregistered [ 1491.682098] Key type ._llcrypt unregistered [ 1499.123444] Key type ._llcrypt registered [ 1499.125308] Key type .llcrypt registered [ 1499.368499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1499.387516] alg: No test for adler32 (adler32-zlib) [ 1500.355335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1500.410301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1500.427028] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1501.350376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1501.405756] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1502.714583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.31@tcp [ 1503.777289] LNetError: Unexpected error -22 connecting to 192.168.202.31@tcp at host 192.168.202.31:988 [ 1509.914546] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:09:37 (1788613777) [ 1510.015481] LNet: 75496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.021769] LNetError: 75496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1510.026295] LNet: Removed LNI 10.1.2.3@tcp [ 1510.036967] LNet: Removed LNI 192.168.202.31@tcp [ 1510.626656] Key type .llcrypt unregistered [ 1510.630941] Key type ._llcrypt unregistered [ 1517.573767] Key type ._llcrypt registered [ 1517.576588] Key type .llcrypt registered [ 1517.823781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1517.862281] alg: No test for adler32 (adler32-zlib) [ 1518.784318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1518.820239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1518.837782] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1519.440656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1519.465325] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 1520.595669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.31@tcp1 --nid 192.168.202.31@tcp2 [ 1525.970942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp1 192.168.202.31@tcp2 [ 1526.008300] LNetError: 76337:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.31@tcp1 to 192.168.202.31@tcp2 - Route table may be misconfigured [ 1526.057894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp2 192.168.202.31@tcp1 [ 1532.589363] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:09:59 (1788613799) [ 1532.861918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1532.886762] LNet: 76910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1532.890359] LNetError: 76910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.920183] LNet: Removed LNI 192.168.202.31@tcp1 [ 1533.928252] LNetError: 76910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.952466] LNet: Removed LNI 192.168.202.31@tcp2 [ 1533.985384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.063932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1534.087122] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1534.910896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1534.928501] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1535.858074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 1535.905405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp 192.168.202.31@tcp [ 1535.939407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp1 192.168.202.31@tcp1 [ 1535.989332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.31@tcp -d 192.168.202.31@tcp -e network_timeout [ 1536.046869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -e network_timeout [ 1536.114169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -e network_timeout [ 1536.176072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -e network_timeout [ 1536.220864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1536.242381] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1536.267352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1536.374848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1536.473096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1537.142810] LNet: 1 local NIs in recovery (showing 1): 192.168.202.31@tcp [ 1543.829140] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:10:11 (1788613811) [ 1543.977828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1544.055145] LNet: 77768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.069630] LNetError: 77768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.091033] LNet: Removed LNI 192.168.202.31@tcp [ 1545.127635] LNet: Removed LNI 192.168.202.31@tcp1 [ 1545.228274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.370858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1545.705461] LNet: 77810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.239672] Key type .llcrypt unregistered [ 1546.253493] Key type ._llcrypt unregistered [ 1558.840885] Key type ._llcrypt registered [ 1558.846115] Key type .llcrypt registered [ 1560.220497] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:10:27 (1788613827) [ 1560.551991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1560.588756] alg: No test for adler32 (adler32-zlib) [ 1561.556931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.240153] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1564.726863] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:10:32 (1788613832) [ 1564.845671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.873595] LNet: 78898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1564.908321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.007292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1565.068264] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1565.727941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1565.743844] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1566.582544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1566.626363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp1 [ 1566.668054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp1 [ 1572.867798] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:10:40 (1788613840) [ 1575.028445] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg231-server [ 1576.889423] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:10:44 (1788613844) [ 1578.911837] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg231-server [ 1580.576915] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:10:47 (1788613847) [ 1581.960627] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg231-server [ 1583.455640] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:10:51 (1788613851) [ 1585.033304] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg231-server [ 1586.668985] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:10:53 (1788613853) [ 1588.556928] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg231-server [ 1590.254286] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:10:57 (1788613857) [ 1592.093953] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg231-server [ 1593.866251] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:11:01 (1788613861) [ 1595.876706] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg231-server [ 1597.411137] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:11:04 (1788613864) [ 1599.617661] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg231-server [ 1601.381319] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:11:08 (1788613868) [ 1603.611352] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg231-server [ 1605.845992] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:11:12 (1788613872) [ 1605.946661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.990521] LNet: 82782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1606.000724] LNetError: 82782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1606.005283] LNet: Removed LNI 192.168.202.31@tcp [ 1607.017318] LNet: Removed LNI 192.168.202.31@tcp1 [ 1607.038587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.095198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1607.122200] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1607.910686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1608.000304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1608.205762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.234234] LNet: 82930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.246849] LNetError: 82930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1608.251091] LNetError: 82930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1608.261595] LNet: Removed LNI 192.168.202.31@tcp [ 1608.280594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.346341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1608.375388] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1609.458849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1609.675219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.687685] LNetError: 83075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.705517] LNet: Removed LNI 192.168.202.31@tcp [ 1609.719351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.774385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.816037] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1610.527958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1610.676580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1610.861212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.884955] LNet: 83222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.890975] LNet: 83222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1611.970040] LNet: Removed LNI 192.168.202.31@tcp [ 1612.016771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.138128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1612.212209] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1612.966337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1613.048267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.064387] LNetError: 83367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1613.068767] LNetError: 83367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1613.077227] LNet: Removed LNI 192.168.202.31@tcp [ 1613.089178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.122215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1613.140517] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1613.717516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1613.878260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1614.126135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.218928] LNet: Removed LNI 192.168.202.31@tcp [ 1615.288523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.453597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1615.508591] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1616.651115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1616.813080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1616.860927] LNet: 83660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.867397] LNet: 83660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1617.895810] LNet: Removed LNI 192.168.202.31@tcp [ 1617.944723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.042233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1618.080808] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1619.296156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1619.423087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1619.755966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.789674] LNetError: 83807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1619.797782] LNetError: 83807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1619.824980] LNet: Removed LNI 192.168.202.31@tcp [ 1619.864621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.013738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1620.077915] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1621.279318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1621.493578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.580397] LNet: Removed LNI 192.168.202.31@tcp [ 1622.599269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.695294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1622.720197] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1623.934802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1624.112985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1624.366571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.446876] LNet: Removed LNI 192.168.202.31@tcp [ 1625.470155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.568436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1625.597127] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1626.739421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1626.909418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.923498] LNet: 84245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.927503] LNet: 84245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1627.947859] LNet: Removed LNI 192.168.202.31@tcp [ 1628.000254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.104930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1628.152608] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1629.340773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1629.445970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1631.085903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.130480] LNetError: 84402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.136572] LNetError: 84402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1632.169050] LNet: Removed LNI 192.168.202.31@tcp [ 1632.219965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.332352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1632.380286] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1633.472650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1635.030192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.134751] LNet: Removed LNI 192.168.202.31@tcp [ 1636.193716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.251233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1636.293247] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1637.009881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1637.086397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.112377] LNet: Removed LNI 192.168.202.31@tcp [ 1637.125958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.194647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1637.209632] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1638.275729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1638.386750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.416223] LNet: Removed LNI 192.168.202.31@tcp [ 1638.429909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.473920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1638.564679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.605403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.683948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1638.781110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.823108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.902817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1639.018506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.069493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.157480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1639.178065] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1639.951599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.974362] LNet: Removed LNI 192.168.202.31@tcp [ 1639.989325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.039369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1648.359763] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:11:55 (1788613915) [ 1648.479078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.563651] LNet: 85573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.570032] LNet: 85573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1648.609586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.671116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1648.690834] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/180] [ 1649.966712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.980473] LNetError: 85717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1649.985737] LNetError: 85717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1649.997490] LNet: Removed LNI 192.168.202.31@tcp231 [ 1650.030733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.104618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1650.128455] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/180] [ 1650.134630] LNet: 85725:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1650.493751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1650.545752] LNet: Removed LNI 192.168.202.31@tcp231 [ 1650.582526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1650.680656] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/180] [ 1652.263973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.300738] LNet: Removed LNI 192.168.202.31@tcp231 [ 1652.319848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.408513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1652.481099] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/0] [ 1652.487866] LNet: 85877:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1652.790817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1652.818960] LNet: Removed LNI 192.168.202.31@tcp231 [ 1652.832622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1652.856239] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/0] [ 1654.110544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.150663] LNet: Removed LNI 192.168.202.31@tcp231 [ 1654.201503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.305544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1654.355127] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/60] [ 1654.365896] LNet: 86029:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1655.271176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1655.304901] LNet: Removed LNI 192.168.202.31@tcp231 [ 1655.320891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1655.370287] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/60] [ 1656.896859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.932369] LNet: Removed LNI 192.168.202.31@tcp231 [ 1656.966517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.047525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1657.082148] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/180] [ 1657.090455] LNet: 86180:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1657.579563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1657.624432] LNet: Removed LNI 192.168.202.31@tcp231 [ 1657.645557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1657.687229] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/180] [ 1667.373321] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:12:14 (1788613934) [ 1667.536066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.603098] LNet: Removed LNI 192.168.202.31@tcp231 [ 1667.658488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.802847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1667.826776] LNet: Added LNI 192.168.202.31@tcp232 [8/256/0/180] [ 1668.418708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1668.445366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1668.476683] LNet: Removed LNI 192.168.202.31@tcp232 [ 1668.491416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1668.522207] LNet: Added LNI 192.168.202.31@tcp232 [8/256/0/180] [ 1669.440100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1669.471551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1669.510435] LNet: Removed LNI 192.168.202.31@tcp232 [ 1669.550368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1669.593896] LNet: Added LNI 192.168.202.31@tcp232 [8/256/0/180] [ 1670.530181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1670.563476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1670.594487] LNet: Removed LNI 192.168.202.31@tcp232 [ 1670.621947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1670.666380] LNet: Added LNI 192.168.202.31@tcp232 [8/256/0/180] [ 1671.523499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1671.582654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1671.621937] LNet: Removed LNI 192.168.202.31@tcp232 [ 1671.640095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1678.874322] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:12:25 (1788613945) [ 1680.033911] Key type .llcrypt unregistered [ 1680.036774] Key type ._llcrypt unregistered [ 1686.502499] Key type ._llcrypt registered [ 1686.504266] Key type .llcrypt registered [ 1686.795576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1686.819439] alg: No test for adler32 (adler32-zlib) [ 1687.821611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1687.892342] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1694.740295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1703.664314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.711976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1703.752882] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1705.760946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_interrupt -r 2 [ 1705.801980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_interrupt -r 2 [ 1705.835725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_interrupt -r 2 [ 1705.872462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_interrupt -r 2 [ 1705.944868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_interrupt -r 2 [ 1705.981261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_interrupt -r 2 [ 1706.022075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_interrupt -r 2 [ 1706.087661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_interrupt -r 2 [ 1706.140645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp 192.168.202.131@tcp [ 1706.171701] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1706.178610] LNet: 1 local NIs in recovery (showing 1): 192.168.202.31@tcp [ 1706.544978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1706.583948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1706.620825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1706.651278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1706.697333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1706.756919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1706.825519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1706.896534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1706.925385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1706.947869] LNet: 88680:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1706.962840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1707.008446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1707.052510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1707.090388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1707.126437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1707.159386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1707.891615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_dropped -r 2 [ 1707.964533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_dropped -r 2 [ 1708.014342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_dropped -r 2 [ 1708.068159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_dropped -r 2 [ 1708.123190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_dropped -r 2 [ 1708.209338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_dropped -r 2 [ 1708.360550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_dropped -r 2 [ 1708.456845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_dropped -r 2 [ 1708.511384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp 192.168.202.131@tcp [ 1708.545217] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1708.882959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1708.928532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1708.963974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1709.003097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1709.046786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1709.100125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1709.144249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1709.200201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1709.240279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1709.276416] LNet: 88803:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1709.301764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1709.331141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1709.392911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1709.450664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1709.498245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1709.542540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1710.221375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_aborted -r 2 [ 1710.260550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_aborted -r 2 [ 1710.325826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_aborted -r 2 [ 1710.386126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_aborted -r 2 [ 1710.438183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_aborted -r 2 [ 1710.520571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_aborted -r 2 [ 1710.598605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_aborted -r 2 [ 1710.662356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_aborted -r 2 [ 1710.757772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp 192.168.202.131@tcp [ 1710.822171] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1711.159031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1711.201630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1711.231960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1711.259162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1711.281648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1711.304200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1711.333896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1711.382699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1711.406137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1711.423510] LNet: 88925:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1711.448630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1711.478450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1711.508446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1711.538951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1711.584103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1711.618752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1712.744198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_no_route -r 2 [ 1712.777690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_no_route -r 2 [ 1712.810552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_no_route -r 2 [ 1712.865823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_no_route -r 2 [ 1712.893423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_no_route -r 2 [ 1712.942826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_no_route -r 2 [ 1713.004724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_no_route -r 2 [ 1713.079756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_no_route -r 2 [ 1713.162649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp 192.168.202.131@tcp [ 1713.200035] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1713.546701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1713.627990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1713.690177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1713.749657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1713.803759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1713.850201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1713.921596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1713.985878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1714.049423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1714.069271] LNet: 89048:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1714.087270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1714.138849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1714.175706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1714.218725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1714.260471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1714.291342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1714.878417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_timeout -r 2 [ 1714.917875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_timeout -r 2 [ 1714.957966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_timeout -r 2 [ 1715.006855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_timeout -r 2 [ 1715.062582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_timeout -r 2 [ 1715.110338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_timeout -r 2 [ 1715.161858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_timeout -r 2 [ 1715.203182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_timeout -r 2 [ 1715.238315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp 192.168.202.131@tcp [ 1715.520911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1715.581869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1715.657853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1715.734883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1715.806831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1715.875992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1715.952177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1716.030314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1716.099062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1716.151220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1716.212997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1716.346935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1716.398373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1716.490942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1716.545986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1717.340627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e remote_dropped -r 2 [ 1717.386601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e remote_dropped -r 2 [ 1717.448372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e remote_dropped -r 2 [ 1717.500623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e remote_dropped -r 2 [ 1717.579165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e remote_dropped -r 2 [ 1717.656455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e remote_dropped -r 2 [ 1717.708609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e remote_dropped -r 2 [ 1717.738194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e remote_dropped -r 2 [ 1717.772986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp 192.168.202.131@tcp [ 1717.796706] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1717.803340] LNet: Skipped 1 previous similar message [ 1717.810429] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.131@tcp [ 1718.057845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1718.091673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1718.153353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1718.205199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1718.265919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1718.330664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1718.372901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1718.445775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1718.525774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1718.568661] LNet: 89292:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1718.583404] LNet: 89292:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1718.614131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1718.717954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1718.823612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1718.874327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1718.928329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1718.985983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1725.270899] LNet: 89367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.278777] LNetError: 89367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.284948] LNet: Removed LNI 192.168.202.31@tcp [ 1725.306784] LNet: Removed LNI 192.168.202.31@tcp1 [ 1725.804740] Key type .llcrypt unregistered [ 1725.810179] Key type ._llcrypt unregistered [ 1743.380376] Key type ._llcrypt registered [ 1743.382627] Key type .llcrypt registered [ 1744.655946] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 09:13:31 (1788614011) [ 1745.130034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1745.145841] alg: No test for adler32 (adler32-zlib) [ 1746.056875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.107841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1746.125533] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1747.122516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1747.162802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.31@tcp [ 1747.204453] LNetError: 90344:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.31@tcp to recovery [ 1747.351285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.31@tcp [ 1747.525769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1754.192169] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 09:13:41 (1788614021) [ 1754.269295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1754.293465] LNet: 90926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1754.304422] LNetError: 90926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.329818] LNet: Removed LNI 192.168.202.31@tcp [ 1754.343745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.443652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1754.462754] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1755.066411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1755.091631] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1755.862472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.31@tcp [ 1755.881176] LNetError: 91199:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.31@tcp to recovery [ 1756.450160] LNet: 1 local NIs in recovery (showing 1): 192.168.202.31@tcp [ 1756.451670] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1757.211907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.31@tcp [ 1766.039197] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 09:13:53 (1788614033) [ 1767.980862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg231-server [ 1770.182949] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 09:13:57 (1788614037) [ 1772.179883] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg231-server [ 1774.117789] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 09:14:01 (1788614041) [ 1776.514624] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg231-server [ 1778.296307] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:14:05 (1788614045) [ 1778.505651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.546564] LNet: 92829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.560197] LNetError: 92829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.566660] LNet: Removed LNI 192.168.202.31@tcp [ 1778.585931] LNet: Removed LNI 192.168.202.31@tcp1 [ 1778.638483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.679481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1778.713966] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1779.748267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1779.855083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1779.891170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1780.013909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1780.042406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1780.121784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1787.126327] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 09:14:14 (1788614054) [ 1787.266025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1787.305695] LNet: 93556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.310186] LNetError: 93556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.314938] LNetError: 93556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1787.333209] LNet: Removed LNI 192.168.202.31@tcp [ 1787.386381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1787.460461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1787.500094] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1794.730768] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:14:21 (1788614061) [ 1795.013668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.061811] LNet: 94264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.067240] LNetError: 94264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.120196] LNet: Removed LNI 192.168.202.31@tcp [ 1795.165871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.254085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1795.284185] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1802.170726] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:14:29 (1788614069) [ 1803.507721] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1805.220161] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:14:32 (1788614072) [ 1805.836920] LNet: 95299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1805.850225] LNetError: 95299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.862440] LNet: Removed LNI 192.168.202.31@tcp [ 1806.681555] Key type .llcrypt unregistered [ 1806.686499] Key type ._llcrypt unregistered [ 1816.040858] Key type ._llcrypt registered [ 1816.042996] Key type .llcrypt registered [ 1816.347604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1816.377349] alg: No test for adler32 (adler32-zlib) [ 1817.391601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1817.482212] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1827.376904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1833.570762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.131@tcp [ 1834.070862] LNet: 95714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.077968] LNetError: 95714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1834.103593] LNet: Removed LNI 192.168.202.31@tcp [ 1834.459437] Key type .llcrypt unregistered [ 1834.462391] Key type ._llcrypt unregistered [ 1846.813781] Key type ._llcrypt registered [ 1846.815985] Key type .llcrypt registered [ 1848.319197] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1850.145750] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1851.824876] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:15:19 (1788614119) [ 1853.441922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1854.979093] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:15:22 (1788614122) [ 1856.966961] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg231-server [ 1858.804795] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 09:15:25 (1788614125) [ 1860.595964] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg231-server [ 1862.272627] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:15:29 (1788614129) [ 1863.712509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1863.976855] alg: No test for adler32 (adler32-zlib) [ 1865.306957] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 1865.669402] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1867.440271] Key type lgssc registered [ 1869.018919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1891.405316] Key type lgssc unregistered [ 1891.698658] LNet: 98886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1891.708927] LNetError: 98886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1891.728348] LNet: Removed LNI 192.168.202.31@tcp [ 1892.715975] Key type .llcrypt unregistered [ 1892.721873] Key type ._llcrypt unregistered [ 1909.210549] Key type ._llcrypt registered [ 1909.214303] Key type .llcrypt registered [ 1910.638593] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 09:16:17 (1788614177) [ 1911.241679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1911.265171] alg: No test for adler32 (adler32-zlib) [ 1912.287272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1912.339186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1912.395610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1912.441951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1912.492898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1920.405978] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 09:16:27 (1788614187) [ 1920.567369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1920.587434] LNet: 100291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.617117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.694796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1920.740041] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1921.643971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1921.681972] LNet: 100441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1921.692770] LNetError: 100441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1921.705317] LNet: Removed LNI 192.168.202.31@tcp [ 1921.722496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.788541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1921.810054] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1922.363281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1922.386519] LNetError: 100469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.404334] LNet: Removed LNI 192.168.202.31@tcp [ 1922.445606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1922.509028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1922.537945] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1931.073099] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 09:16:38 (1788614198) [ 1931.272363] LNet: 101060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.275170] LNet: 101060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1931.278258] LNetError: 101060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.292605] LNet: Removed LNI 192.168.202.31@tcp [ 1931.980626] Key type .llcrypt unregistered [ 1931.982760] Key type ._llcrypt unregistered [ 1938.073548] Key type ._llcrypt registered [ 1938.077497] Key type .llcrypt registered [ 1938.343455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1938.368307] alg: No test for adler32 (adler32-zlib) [ 1939.309894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1939.380717] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1939.384083] LNet: *** cfs_fail_loc=e003, val=100*** [ 1940.045336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1940.066353] LNet: *** cfs_fail_loc=e003, val=100*** [ 1940.069737] LNet: Skipped 2 previous similar messages [ 1940.074926] Ping reply received with mapping: [ 1940.077148] found metadata reply NI[192.168.202.31@tcp] [ 1940.079465] 1f ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1940.082703] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1940.086716] 5a 5a 5a 5a 5a 5a 5a 5a [ 1940.202660] LNet: 101454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1940.211224] LNetError: 101454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1940.231327] LNet: Removed LNI 192.168.202.31@tcp [ 1940.869445] Key type .llcrypt unregistered [ 1940.873187] Key type ._llcrypt unregistered [ 1954.095346] Key type ._llcrypt registered [ 1954.105115] Key type .llcrypt registered [ 1955.337471] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:17:02 (1788614222) [ 1955.890403] Key type .llcrypt unregistered [ 1955.892746] Key type ._llcrypt unregistered [ 1962.715221] Key type ._llcrypt registered [ 1962.717418] Key type .llcrypt registered [ 1962.891678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1962.907535] alg: No test for adler32 (adler32-zlib) [ 1964.071675] LNetError: 102491:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1964.505420] Key type .llcrypt unregistered [ 1964.508088] Key type ._llcrypt unregistered [ 1969.613620] Key type ._llcrypt registered [ 1969.615482] Key type .llcrypt registered [ 1970.919327] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 09:17:18 (1788614238) [ 1971.440380] Key type .llcrypt unregistered [ 1971.442043] Key type ._llcrypt unregistered [ 1978.072622] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1978.079087] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1980.699648] Key type ._llcrypt registered [ 1980.707163] Key type .llcrypt registered [ 1980.956603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1980.975787] alg: No test for adler32 (adler32-zlib) [ 1981.904966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.986585] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1992.096628] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 09:17:39 (1788614259) [ 1992.270904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.308943] LNet: 104259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.326679] LNetError: 104259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1992.363688] LNet: Removed LNI 10.1.2.3@tcp [ 1992.401507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.255815] Key type .llcrypt unregistered [ 1993.259543] Key type ._llcrypt unregistered [ 2000.111283] Key type ._llcrypt registered [ 2000.119276] Key type .llcrypt registered [ 2000.383598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2000.405839] alg: No test for adler32 (adler32-zlib) [ 2001.402320] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2001.488133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2001.653971] LNet: 104549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2001.658978] LNetError: 104549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2002.745582] LNet: Removed LNI 192.168.202.31@tcp [ 2003.411646] Key type .llcrypt unregistered [ 2003.415457] Key type ._llcrypt unregistered [ 2015.588631] Key type ._llcrypt registered [ 2015.591390] Key type .llcrypt registered [ 2016.851502] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:18:04 (1788614284) [ 2017.660438] Key type .llcrypt unregistered [ 2017.663456] Key type ._llcrypt unregistered [ 2023.869182] Key type ._llcrypt registered [ 2023.870649] Key type .llcrypt registered [ 2024.178977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2024.206859] alg: No test for adler32 (adler32-zlib) [ 2031.696677] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:18:18 (1788614298) [ 2031.848537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2031.903289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.012874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2032.058768] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2032.820535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2032.852756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2032.880597] LNet: Removed LNI 192.168.202.31@tcp [ 2033.180956] LNet: 106288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.520452] Key type .llcrypt unregistered [ 2033.526630] Key type ._llcrypt unregistered [ 2045.773081] Key type ._llcrypt registered [ 2045.774424] Key type .llcrypt registered [ 2047.378745] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:18:34 (1788614314) [ 2047.864526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2047.885247] alg: No test for adler32 (adler32-zlib) [ 2048.896241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2049.002852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2049.045036] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2049.878265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2049.962968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.31@tcp [ 2049.996930] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2050.001618] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2050.007478] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2049, 0, 0 [ 2050.013768] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2050.016794] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2050.019465] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2050.023379] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2049, 0, 0 [ 2050.030965] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2050.037210] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2050.042741] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2050.046705] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2049, 0, 2099 [ 2050.053621] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2050.056282] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2050.061286] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2050.065901] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2049, 0, 2099 [ 2050.072473] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2057.563429] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:18:44 (1788614324) [ 2058.197207] LNet: 107826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.204203] LNetError: 107826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2059.240108] LNet: Removed LNI 192.168.202.31@tcp [ 2060.079685] Key type .llcrypt unregistered [ 2060.084345] Key type ._llcrypt unregistered [ 2068.264311] Key type ._llcrypt registered [ 2068.266849] Key type .llcrypt registered [ 2068.465419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2068.494723] alg: No test for adler32 (adler32-zlib) [ 2069.551844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2069.573049] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2076.774817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 2085.887557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2085.947462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2085.994427] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 2087.088151] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2087.215968] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2087.224931] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2087.232153] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2089.581714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2089.606686] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2101.739737] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:19:28 (1788614368) [ 2101.985295] LNet: 109333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.998026] LNetError: 109333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.012816] LNet: Removed LNI 192.168.202.31@tcp [ 2103.072669] LNet: Removed LNI 192.168.202.31@tcp1 [ 2103.078637] LNetError: 109333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2103.102713] LNetError: 109333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2103.125775] LNet: Removed LNI 10.1.2.3@tcp99 [ 2104.135932] Key type .llcrypt unregistered [ 2104.139539] Key type ._llcrypt unregistered [ 2111.559111] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2111.574062] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2114.130363] Key type ._llcrypt registered [ 2114.131986] Key type .llcrypt registered [ 2114.388281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2114.412622] alg: No test for adler32 (adler32-zlib) [ 2115.410522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2115.463492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2115.482154] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2116.445306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2116.467234] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2119.029969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2119.081784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2119.140546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2119.163949] LNetError: 110076:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2119.220806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2119.265833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2119.904621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2120.025797] LNet: 110096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.036304] LNetError: 110096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2120.051917] LNet: Removed LNI 10.1.2.3@tcp [ 2121.129376] LNet: Removed LNI 192.168.202.31@tcp [ 2121.940424] Key type .llcrypt unregistered [ 2121.941903] Key type ._llcrypt unregistered [ 2134.713863] Key type ._llcrypt registered [ 2134.716701] Key type .llcrypt registered [ 2136.136827] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:20:03 (1788614403) [ 2136.450588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2136.464339] alg: No test for adler32 (adler32-zlib) [ 2137.438384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.515757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2137.532552] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2138.230326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2138.309509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg231-client@tcp [ 2144.940932] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:20:11 (1788614411) [ 2145.058162] LNet: 111617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2145.070164] LNetError: 111617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2145.089376] LNet: Removed LNI 192.168.202.31@tcp [ 2145.837815] Key type .llcrypt unregistered [ 2145.841383] Key type ._llcrypt unregistered [ 2153.740800] Key type ._llcrypt registered [ 2153.752755] Key type .llcrypt registered [ 2154.042552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2154.096812] alg: No test for adler32 (adler32-zlib) [ 2155.019931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2155.047774] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2155.086344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2161.766811] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:20:28 (1788614428) [ 2163.378798] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2165.081684] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:20:32 (1788614432) [ 2166.443742] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2168.091774] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 09:20:35 (1788614435) [ 2168.266281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2168.286959] LNet: 113051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.292866] LNetError: 113051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.302399] LNet: Removed LNI 192.168.202.31@tcp [ 2168.314858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.389911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2168.417969] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2169.143724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2169.188415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2176.495401] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:20:43 (1788614443) [ 2176.888933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2176.913802] LNet: 113760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2176.919963] LNetError: 113760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2176.935910] LNet: Removed LNI 192.168.202.31@tcp [ 2176.960570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2177.059928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2177.107296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2177.892780] Key type .llcrypt unregistered [ 2177.895946] Key type ._llcrypt unregistered [ 2189.424467] Key type ._llcrypt registered [ 2189.425622] Key type .llcrypt registered [ 2190.585436] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:20:57 (1788614457) [ 2191.047485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2191.065376] alg: No test for adler32 (adler32-zlib) [ 2191.977488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2192.006651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2192.018495] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2192.951123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2193.079087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2200.392678] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:21:07 (1788614467) [ 2200.684228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2200.723938] LNet: 115313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.732981] LNetError: 115313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2200.750857] LNet: Removed LNI 192.168.202.31@tcp [ 2200.770176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2200.835710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2200.909286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2208.554614] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 09:21:15 (1788614475) [ 2208.725961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2208.796091] LNet: 115891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2208.883325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2208.997787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2209.527955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2209.563548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2209.632301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2210.270466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2210.335845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2210.467460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2210.491208] LNet: 115927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2210.517943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.596738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2211.198376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2211.246575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2211.309583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2211.706587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2211.727596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2218.980056] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 09:21:25 (1788614485) [ 2219.167606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2219.207697] LNet: 116526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.237697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.288556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2219.312955] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2220.020837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2220.036363] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 2220.966224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2221.027759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2221.244375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2221.261631] LNetError: 116806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.304188] LNet: Removed LNI 192.168.202.31@tcp [ 2222.306121] LNetError: 116806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.337435] LNet: Removed LNI 192.168.202.31@tcp2 [ 2222.383829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.520742] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2222.531795] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 2222.546393] LNet: 116812:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2223.457631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2223.728197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2223.784689] LNet: 116849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2223.808371] LNet: 116849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2224.864206] LNet: Removed LNI 192.168.202.31@tcp [ 2224.874386] LNetError: 116849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2224.884864] LNetError: 116849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2224.904829] LNet: Removed LNI 192.168.202.31@tcp2 [ 2224.962607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.040748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2225.130381] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2226.116400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2226.135316] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 2227.078699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2227.107961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2227.154639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2227.406417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2228.450552] LNet: Removed LNI 192.168.202.31@tcp [ 2229.519195] LNet: Removed LNI 192.168.202.31@tcp2 [ 2229.548785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2229.644292] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2229.655246] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 2229.672993] LNet: 117140:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2230.548364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2230.889915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2230.945337] LNetError: 117177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2230.952358] LNetError: 117177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2230.958923] LNet: Removed LNI 192.168.202.31@tcp [ 2231.974941] LNet: Removed LNI 192.168.202.31@tcp2 [ 2232.025710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.154940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2232.197305] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2233.079217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2233.111553] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 2234.149928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2234.220572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2234.445261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.476146] LNet: 117457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.488526] LNet: 117457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2235.553494] LNet: Removed LNI 192.168.202.31@tcp [ 2235.576213] LNet: Removed LNI 192.168.202.31@tcp2 [ 2235.625956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.720133] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2235.741764] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 2235.781210] LNet: 117463:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2236.501827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2236.711459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2237.792193] LNet: Removed LNI 192.168.202.31@tcp [ 2237.808061] LNet: Removed LNI 192.168.202.31@tcp2 [ 2237.862273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2237.930159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2237.970164] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2238.946415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2238.973870] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 2239.866443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2239.903147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2239.951501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2240.173432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2240.201454] LNetError: 117785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2240.209494] LNetError: 117785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2241.254101] LNet: Removed LNI 192.168.202.31@tcp [ 2242.278837] LNet: Removed LNI 192.168.202.31@tcp2 [ 2242.317406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2242.382488] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2242.387302] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 2242.393855] LNet: 117791:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2242.996548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2243.214189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2243.256459] LNet: Removed LNI 192.168.202.31@tcp [ 2244.326961] LNet: Removed LNI 192.168.202.31@tcp2 [ 2244.365489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.503439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2244.540588] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2245.451424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2245.490944] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 2246.345281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2246.374996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2246.440722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2246.942223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2247.969775] LNet: Removed LNI 192.168.202.31@tcp [ 2247.985594] LNet: Removed LNI 192.168.202.31@tcp2 [ 2248.013785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.090925] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2248.098884] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 2248.103699] LNet: 118131:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2248.661868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2248.923287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2249.952287] LNet: Removed LNI 192.168.202.31@tcp [ 2249.968473] LNet: Removed LNI 192.168.202.31@tcp2 [ 2249.993908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2250.078889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2250.115995] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2250.878875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2250.902514] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 2251.795300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2251.857776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2251.895545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2251.947906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2252.237845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2252.253348] LNet: 118485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.257534] LNet: 118485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2253.284091] LNet: Removed LNI 192.168.202.31@tcp [ 2254.323797] LNet: Removed LNI 192.168.202.31@tcp2 [ 2254.373818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.438889] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2254.445296] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 2254.873581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2262.300247] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2264.036939] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 09:22:11 (1788614531) [ 2264.128374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2264.153533] LNetError: 119244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2264.162092] LNetError: 119244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2264.170280] LNet: Removed LNI 192.168.202.31@tcp [ 2265.199764] LNet: Removed LNI 192.168.202.31@tcp2 [ 2265.253067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.591193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2265.616968] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2266.126652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2266.144566] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2267.233380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2267.264449] LNet: Removed LNI 10.1.2.3@tcp [ 2267.286043] LNet: Removed LNI 192.168.202.31@tcp [ 2267.308606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2267.338600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2267.356605] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2267.365284] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2267.582904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2267.598609] LNet: Removed LNI 192.168.202.31@tcp [ 2267.608981] LNet: Removed LNI 10.1.2.3@tcp [ 2273.399274] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 09:22:20 (1788614540) [ 2273.759989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.825632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.931055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2273.973975] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2274.815758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2274.847954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2274.873099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2274.918762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2274.943883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2274.969686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2275.020769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2275.050148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2275.070815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2275.094433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2275.162032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2275.237348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2275.418484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2282.357350] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 09:22:30 (1788614550) [ 2282.884951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2283.943251] LNet: Removed LNI 192.168.202.31@tcp [ 2284.002194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.143621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2284.168676] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2285.099796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2285.139694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2285.185725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2285.241806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2285.283077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2285.311663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2285.356786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2285.413627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2285.531772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2285.588481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2285.652788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2285.697459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2285.931308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2286.007044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2286.072762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2286.138097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2286.196199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2286.275328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2286.346214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2286.416230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2286.440660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2286.462458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2286.505970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 2293.497229] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:22:40 (1788614560) [ 2293.606449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2293.627298] LNet: 121613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.632431] LNet: 121613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2294.699748] LNet: Removed LNI 192.168.202.31@tcp [ 2294.779391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2294.954645] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2295.156535] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2295.162448] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2295.166560] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2297.527511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2297.572213] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2298.398968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2298.434780] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2299.930394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2299.978752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2300.017470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.31@tcp -m PUT -e local_timeout -r 1 [ 2300.059655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2300.120927] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2300.124642] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2300.498724] LNetError: 121681:0:(lib-move.c:4794:lnet_parse()) 192.168.202.31@tcp, src 192.168.202.31@tcp: Dropping PUT (error -108 looking up sender) [ 2301.550165] LNetError: 122140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2301.563654] LNetError: 122140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2301.574763] LNet: Removed LNI 10.1.2.3@tcp [ 2302.638816] LNet: Removed LNI 192.168.202.31@tcp [ 2303.023572] Key type .llcrypt unregistered [ 2303.025243] Key type ._llcrypt unregistered [ 2313.827822] Key type ._llcrypt registered [ 2313.829485] Key type .llcrypt registered [ 2315.088867] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:23:02 (1788614582) [ 2315.456887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2315.473780] alg: No test for adler32 (adler32-zlib) [ 2316.386633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.705649] LNet: 122989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.030334] Key type .llcrypt unregistered [ 2317.032333] Key type ._llcrypt unregistered [ 2321.860322] Key type ._llcrypt registered [ 2321.862990] Key type .llcrypt registered [ 2322.050311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2322.064555] alg: No test for adler32 (adler32-zlib) [ 2322.941609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2322.963572] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2330.759573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 2330.945645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2331.489392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.131@tcp [ 2331.829139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2331.866434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2331.908679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2331.939858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2331.966594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2332.002970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2332.043857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2332.077764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2332.110300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2332.143639] LNet: 123437:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2332.172260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2332.198960] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.131@tcp [ 2332.239222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2332.283955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2332.326702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2332.362597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2332.411623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2333.247853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2333.624433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2333.660576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2333.703497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2333.741747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2333.777456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2333.822159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2333.877780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2333.948194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2334.015289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2334.039389] LNet: 123556:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2334.064351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2334.127395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2334.159854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2334.194717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2334.226870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2334.266845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2335.020431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.131@tcp [ 2335.282883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2335.322468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2335.355787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2335.390970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2335.431715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2335.467792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2335.508386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2335.546915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2335.574375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2335.615843] LNet: 123676:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2335.650878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2335.694635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2335.725545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2335.752820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2335.831608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2335.899507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2336.867703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2337.271368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2337.324469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2337.369547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2337.394517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2337.440916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2337.474733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2337.529172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2337.592508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2337.644648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2337.663661] LNet: 123795:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2337.700470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2337.736631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2337.806306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2337.879404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2337.921329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2337.984373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2338.372944] LNet: 123841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2338.381050] LNetError: 123841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2339.436482] LNet: Removed LNI 192.168.202.31@tcp [ 2339.998682] Key type .llcrypt unregistered [ 2340.000942] Key type ._llcrypt unregistered [ 2359.629118] Key type ._llcrypt registered [ 2359.633440] Key type .llcrypt registered [ 2360.979538] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 09:23:48 (1788614628) [ 2361.606554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2361.624763] alg: No test for adler32 (adler32-zlib) [ 2362.609930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2363.208711] LNet: 124743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2364.072566] Key type .llcrypt unregistered [ 2364.076370] Key type ._llcrypt unregistered [ 2371.242406] Key type ._llcrypt registered [ 2371.243905] Key type .llcrypt registered [ 2371.427574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2371.462585] alg: No test for adler32 (adler32-zlib) [ 2372.398576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2372.421437] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2380.452288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 2389.313249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2389.416786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2389.460213] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 2390.528426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2391.186545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.131@tcp [ 2391.467417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2391.496257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2391.524760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2391.551826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2391.575763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2391.599454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2391.621654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2391.649472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2391.673512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2391.686880] LNet: 125394:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2391.706697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2391.730463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2391.754627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2391.789508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2391.819939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2391.850990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2392.397975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2392.694934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2392.744201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2392.814813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2392.860755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2392.889655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2392.929856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2392.933533] LNet: 2 peer NIs in recovery (showing 2): 192.168.202.131@tcp, 192.168.202.131@tcp1 [ 2392.971943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2393.022646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2393.060885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2393.091496] LNet: 125513:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2393.116443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2393.159376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2393.191687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2393.253548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2393.287716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2393.327903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2394.067221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.131@tcp [ 2394.346674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2394.371760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2394.401475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2394.428482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2394.460557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2394.487129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2394.514573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2394.541768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2394.573177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2394.589551] LNet: 125632:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2394.601717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2394.621725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2394.646445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2394.671823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2394.703368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2394.737942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2395.412692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2395.682687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2395.714810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2395.746233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2395.786514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2395.827224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2395.861760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2395.896601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2395.938907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2395.964981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2396.013037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2396.037697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2396.068970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2396.105945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2396.128698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2396.153584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2401.432735] LNet: 125833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2401.437654] LNetError: 125833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2401.443837] LNet: Removed LNI 192.168.202.31@tcp [ 2401.474637] LNet: Removed LNI 192.168.202.31@tcp1 [ 2401.949942] Key type .llcrypt unregistered [ 2401.957770] Key type ._llcrypt unregistered [ 2421.519322] Key type ._llcrypt registered [ 2421.521480] Key type .llcrypt registered [ 2422.637655] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 09:24:50 (1788614690) [ 2423.298482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2423.314435] alg: No test for adler32 (adler32-zlib) [ 2424.334979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2424.405747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2424.445046] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2425.494502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2425.514780] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 2426.397186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 2426.466750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2427.552364] LNet: Removed LNI 192.168.202.31@tcp [ 2433.976837] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 09:25:01 (1788614701) [ 2435.872555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2438.003770] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 09:25:04 (1788614704) [ 2445.356905] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 09:25:12 (1788614712) [ 2453.753178] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 09:25:20 (1788614720) [ 2461.184891] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 09:25:28 (1788614728) [ 2468.239078] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 09:25:35 (1788614735) [ 2476.560807] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 09:25:43 (1788614743) [ 2483.790768] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 09:25:51 (1788614751) [ 2492.394554] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 09:25:59 (1788614759) [ 2501.104620] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 09:26:08 (1788614768) [ 2509.534917] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 09:26:16 (1788614776) [ 2516.723565] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 09:26:23 (1788614783) [ 2525.188071] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 09:26:32 (1788614792) [ 2533.142786] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 09:26:40 (1788614800) [ 2540.685503] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 09:26:47 (1788614807) [ 2548.733817] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 09:26:55 (1788614815) [ 2556.545347] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 09:27:03 (1788614823) [ 2565.217209] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 09:27:12 (1788614832) [ 2572.664486] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 09:27:19 (1788614839) [ 2572.763385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2572.806638] LNet: 137530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2572.823219] LNetError: 137530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2572.864454] LNet: Removed LNI 192.168.202.31@tcp2 [ 2572.887364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2573.029913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2573.049163] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2574.278273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2574.306934] LNet: 137674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2574.313294] LNetError: 137674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2574.329045] LNet: Removed LNI 192.168.202.31@tcp [ 2574.374605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2574.463812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2574.494861] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2581.595676] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 09:27:28 (1788614848) [ 2588.896460] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 09:27:36 (1788614856) [ 2597.907973] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 09:27:44 (1788614864) [ 2605.757364] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 09:27:52 (1788614872) [ 2613.856465] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 09:28:00 (1788614880) [ 2621.313663] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 09:28:08 (1788614888) [ 2629.636470] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 09:28:16 (1788614896) [ 2639.135650] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 09:28:25 (1788614905) [ 2648.017477] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 09:28:35 (1788614915) [ 2656.424843] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 09:28:43 (1788614923) [ 2664.137410] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 09:28:51 (1788614931) [ 2672.207258] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 09:28:59 (1788614939) [ 2680.103422] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2427 sec ========== 09:29:06 (1788614946) [ 2680.453964] LNet: 145142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2680.461857] LNetError: 145142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2680.488855] LNet: Removed LNI 192.168.202.31@tcp [ 2681.080554] Key type .llcrypt unregistered [ 2681.090985] Key type ._llcrypt unregistered [ 2693.135566] Key type ._llcrypt registered [ 2693.141512] Key type .llcrypt registered [ 2693.796990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2693.810256] alg: No test for adler32 (adler32-zlib) [ 2695.123702] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 2695.530217] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2697.352179] Key type lgssc registered [ 2698.682643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2763.262258] Lustre: Mounted lustre-client - version 2.17.58_2_ge9089f3 [ 2768.855891] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2776.644675] Lustre: Unmounted lustre-client [ 2813.998556] Key type lgssc unregistered [ 2814.334494] LNet: 148091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2814.346529] LNetError: 148091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2814.375103] LNet: Removed LNI 192.168.202.31@tcp [ 2815.024236] Key type .llcrypt unregistered [ 2815.030719] Key type ._llcrypt unregistered