[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 806405895 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003063] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.007730] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.010115] LSM: Security Framework initializing [ 0.011041] Yama: becoming mindful. [ 0.012026] SELinux: Initializing. [ 0.013058] *** VALIDATE selinux *** [ 0.021453] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025619] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028120] *** VALIDATE tmpfs *** [ 0.029413] *** VALIDATE proc *** [ 0.031210] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033239] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034137] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036024] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.039772] debug: unmapping init [mem 0xffffffffb6e59000-0xffffffffb6e60fff] [ 0.041228] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042668] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043016] ... version: 2 [ 0.044010] ... bit width: 48 [ 0.045008] ... generic registers: 4 [ 0.046010] ... value mask: 0000ffffffffffff [ 0.047009] ... max period: 00007fffffffffff [ 0.048009] ... fixed-purpose events: 3 [ 0.049008] ... event mask: 000000070000000f [ 0.050282] rcu: Hierarchical SRCU implementation. [ 0.052484] smp: Bringing up secondary CPUs ... [ 0.053545] x86: Booting SMP configuration: [ 0.054019] .... node #0, CPUs: #1 #2 #3 [ 0.058623] smp: Brought up 1 node, 4 CPUs [ 0.060011] smpboot: Max logical packages: 1 [ 0.061010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138263] node 0 deferred pages initialised in 74ms [ 0.142284] devtmpfs: initialized [ 0.143298] x86/mm: Memory block size: 128MB [ 0.146039] gcov: version magic: 0x41383552 [ 0.148261] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150316] pinctrl core: initialized pinctrl subsystem [ 0.151228] [ 0.151745] ************************************************************* [ 0.152011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153016] ** ** [ 0.154016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155008] ** ** [ 0.156010] ** This means that this kernel is built to expose internal ** [ 0.157009] ** IOMMU data structures, which may compromise security on ** [ 0.158068] ** your system. ** [ 0.159009] ** ** [ 0.160008] ** If you see this message and you are not debugging the ** [ 0.161009] ** kernel, report this immediately to your vendor! ** [ 0.162008] ** ** [ 0.163011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164024] ************************************************************* [ 0.165816] NET: Registered protocol family 16 [ 0.166508] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.167055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.168051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.169656] cpuidle: using governor menu [ 0.172084] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.173381] PCI: Using configuration type 1 for base access [ 0.174123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.182043] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.183062] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.184497] cryptd: max_cpu_qlen set to 1000 [ 0.185291] ACPI: Added _OSI(Module Device) [ 0.186000] ACPI: Added _OSI(Processor Device) [ 0.186000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.188038] ACPI: Added _OSI(Processor Aggregator Device) [ 0.192845] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.196697] ACPI: Interpreter enabled [ 0.197047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.198007] ACPI: Using IOAPIC for interrupt routing [ 0.199081] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.200421] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.211552] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.212051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.213015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.214069] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.216444] acpiphp: Slot [2] registered [ 0.217082] acpiphp: Slot [5] registered [ 0.218093] acpiphp: Slot [6] registered [ 0.219104] acpiphp: Slot [3] registered [ 0.220077] acpiphp: Slot [4] registered [ 0.221135] acpiphp: Slot [7] registered [ 0.222078] acpiphp: Slot [8] registered [ 0.223152] acpiphp: Slot [9] registered [ 0.224070] acpiphp: Slot [10] registered [ 0.225093] acpiphp: Slot [11] registered [ 0.226127] acpiphp: Slot [12] registered [ 0.227072] acpiphp: Slot [13] registered [ 0.228070] acpiphp: Slot [14] registered [ 0.229072] acpiphp: Slot [15] registered [ 0.230076] acpiphp: Slot [16] registered [ 0.231073] acpiphp: Slot [17] registered [ 0.232147] acpiphp: Slot [18] registered [ 0.233127] acpiphp: Slot [19] registered [ 0.234090] acpiphp: Slot [20] registered [ 0.235122] acpiphp: Slot [21] registered [ 0.236084] acpiphp: Slot [22] registered [ 0.237154] acpiphp: Slot [23] registered [ 0.238085] acpiphp: Slot [24] registered [ 0.239100] acpiphp: Slot [25] registered [ 0.240114] acpiphp: Slot [26] registered [ 0.241077] acpiphp: Slot [27] registered [ 0.242074] acpiphp: Slot [28] registered [ 0.243083] acpiphp: Slot [29] registered [ 0.244095] acpiphp: Slot [30] registered [ 0.245158] acpiphp: Slot [31] registered [ 0.246057] PCI host bridge to bus 0000:00 [ 0.247018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.248015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.249016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.250021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.251015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.252023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.253167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.255025] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.257198] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.262013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.264599] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.265013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.266012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.267018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.268562] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.270192] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.271032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.272796] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.274860] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.282014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.284012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.288427] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.299012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.307011] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.327016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.340000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.347010] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.370827] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.429014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.442000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.445303] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.447477] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.450309] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.452384] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.457052] iommu: Default domain type: Passthrough [ 0.458000] SCSI subsystem initialized [ 0.458000] ACPI: bus type USB registered [ 0.460103] usbcore: registered new interface driver usbfs [ 0.462050] usbcore: registered new interface driver hub [ 0.464091] usbcore: registered new device driver usb [ 0.465199] pps_core: LinuxPPS API ver. 1 registered [ 0.467007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.469053] PTP clock support registered [ 0.470426] EDAC MC: Ver: 3.0.0 [ 0.471611] PCI: Using ACPI for IRQ routing [ 0.472000] NetLabel: Initializing [ 0.472067] NetLabel: domain hash size = 128 [ 0.473009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.475110] NetLabel: unlabeled traffic allowed by default [ 0.477263] vgaarb: loaded [ 0.478589] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.480054] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.487573] clocksource: Switched to clocksource kvm-clock [ 0.601115] VFS: Disk quotas dquot_6.6.0 [ 0.602421] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.604632] *** VALIDATE ramfs *** [ 0.605674] *** VALIDATE hugetlbfs *** [ 0.607633] pnp: PnP ACPI init [ 0.609769] pnp: PnP ACPI: found 6 devices [ 0.625221] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.628231] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.630442] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.632309] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.634324] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.636348] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.638716] NET: Registered protocol family 2 [ 0.640803] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.645513] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.648548] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.653941] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.657723] TCP: Hash tables configured (established 65536 bind 65536) [ 0.660889] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.664235] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.667301] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.670453] NET: Registered protocol family 1 [ 0.673325] RPC: Registered named UNIX socket transport module. [ 0.678911] RPC: Registered udp transport module. [ 0.681223] RPC: Registered tcp transport module. [ 0.683182] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.684958] NET: Registered protocol family 44 [ 0.686441] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.688586] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.690698] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.692866] PCI: CLS 0 bytes, default 64 [ 0.694563] Unpacking initramfs... [ 2.502395] debug: unmapping init [mem 0xffff9328bcc64000-0xffff9328bffcffff] [ 2.506351] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.513063] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.523617] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.164335] Initialise system trusted keyrings [ 4.165857] Key type blacklist registered [ 4.168505] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.176804] zbud: loaded [ 4.181241] *** VALIDATE nfs *** [ 4.182330] *** VALIDATE nfs4 *** [ 4.183552] pstore: using deflate compression [ 4.186712] Platform Keyring initialized [ 4.340495] NET: Registered protocol family 38 [ 4.343204] Key type asymmetric registered [ 4.344940] Asymmetric key parser 'x509' registered [ 4.347514] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.352443] io scheduler mq-deadline registered [ 4.353885] io scheduler kyber registered [ 4.356296] io scheduler bfq registered [ 4.358952] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.362687] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.367702] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.371544] ACPI: Power Button [PWRF] [ 4.384705] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.393710] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.415693] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.444334] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.473205] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.479468] Non-volatile memory driver v1.3 [ 4.480765] Linux agpgart interface v0.103 [ 4.515722] virtio_blk virtio1: [vda] 146216 512-byte logical blocks (74.9 MB/71.4 MiB) [ 4.520066] vda: detected capacity change from 0 to 74862592 [ 4.533146] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.535887] vdb: detected capacity change from 0 to 1073741824 [ 4.541786] libphy: Fixed MDIO Bus: probed [ 4.547455] usbcore: registered new interface driver usbserial_generic [ 4.550095] usbserial: USB Serial support registered for generic [ 4.552480] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.557630] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.567175] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.569901] mousedev: PS/2 mouse device common for all mice [ 4.574745] rtc_cmos 00:05: RTC can wake from S4 [ 4.577771] rtc_cmos 00:05: registered as rtc0 [ 4.580108] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.583055] intel_pstate: CPU model not supported [ 4.585239] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.588894] hid: raw HID events driver (C) Jiri Kosina [ 4.592780] usbcore: registered new interface driver usbhid [ 4.595355] usbhid: USB HID core driver [ 4.653579] drop_monitor: Initializing network drop monitor service [ 4.672670] Initializing XFRM netlink socket [ 4.691048] NET: Registered protocol family 10 [ 4.810666] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.834727] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.960333] Segment Routing with IPv6 [ 4.962054] NET: Registered protocol family 17 [ 4.965888] mpls_gso: MPLS GSO support [ 4.977922] RAS: Correctable Errors collector initialized. [ 4.997168] AVX version of gcm_enc/dec engaged. [ 5.001907] AES CTR mode by8 optimization enabled [ 5.309091] sched_clock: Marking stable (5309067065, 0)->(6477922118, -1168855053) [ 5.330694] registered taskstats version 1 [ 5.343687] Loading compiled-in X.509 certificates [ 5.353899] zswap: loaded using pool lzo/zbud [ 5.419142] Key type big_key registered [ 5.431588] Key type encrypted registered [ 5.433198] ima: No TPM chip found, activating TPM-bypass! [ 5.435820] ima: Allocated hash algorithm: sha1 [ 5.439290] ima: No architecture policies found [ 5.441453] evm: Initialising EVM extended attributes: [ 5.443569] evm: security.selinux [ 5.444855] evm: security.ima [ 5.446089] evm: security.capability [ 5.447521] evm: HMAC attrs: 0x1 [ 5.449734] rtc_cmos 00:05: setting system clock to 2026-08-24 05:09:19 UTC (1787548159) [ 5.456513] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 5.459677] debug: unmapping init [mem 0xffffffffb6b82000-0xffffffffb6e58fff] [ 5.467241] Write protecting the kernel read-only data: 28672k [ 5.471183] debug: unmapping init [mem 0xffffffffb5203000-0xffffffffb53fffff] [ 5.474767] debug: unmapping init [mem 0xffffffffb5b14000-0xffffffffb5bfffff] [ 5.511465] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.520482] systemd[1]: Detected virtualization kvm. [ 5.522693] systemd[1]: Detected architecture x86-64. [ 5.525336] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.595593] systemd[1]: No hostname configured. [ 5.597776] systemd[1]: Set hostname to . [ 5.600333] random: systemd: uninitialized urandom read (16 bytes read) [ 5.603174] systemd[1]: Initializing machine ID from random generator. [ 6.081789] random: systemd: uninitialized urandom read (16 bytes read) [ 6.087455] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 6.104619] random: systemd: uninitialized urandom read (16 bytes read) [ 6.114944] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.130657] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.430504] device-mapper: uevent: version 1.0.3 [ 8.433270] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 9.413644] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Mounting Kernel Configuration File System... [ 9.653164] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Mounted Kernel Configuration File System. [ 12.273884] scsi host0: ata_piix [ 12.313098] scsi host1: ata_piix [ 12.314771] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 12.317365] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 15.074535] random: crng init done [ 15.077528] random: 7 urandom warning(s) missed due to ratelimiting [ 18.399066] dracut-initqueue[574]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.585463] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.996620] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.355872] SELinux: Disabled at runtime. [ 23.538712] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.548486] systemd[1]: Detected virtualization kvm. [ 23.549804] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.331844] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.335771] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.369810] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.415398] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.426511] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.492740] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.522789] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK [0[ 26.130965] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started Flush Journal to Persistent Storage. [ 27.795374] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.369704] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.539700] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.916499] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.979331] EDAC sbridge: Ver: 1.1.2 [ 33.058588] Key type dns_resolver registered [ 33.678147] NFS: Registering the id_resolver key type [ 33.682440] Key type id_resolver registered [ 33.686368] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg324-client login: [ 66.785336] hrtimer: interrupt took 8421138 ns [ 88.596740] libcfs: loading out-of-tree module taints kernel. [ 88.625854] Key type ._llcrypt registered [ 88.632372] Key type .llcrypt registered [ 89.085786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 89.094609] alg: No test for adler32 (adler32-zlib) [ 90.301368] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 90.870444] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 92.536432] Key type lgssc registered [ 93.901698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 232.477796] Lustre: Mounted lustre-client - version 2.17.57_83_g6b3cfd8 [ 237.899156] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 254.257050] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing check_logdir /tmp/testlogs/ [ 258.023432] Lustre: lustre-OST0000-osc-ffff9329137dd800: disconnect after 24s idle [ 259.759394] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing yml_node [ 264.443087] Lustre: DEBUG MARKER: Client: 2.17.57.83 [ 267.113654] Lustre: DEBUG MARKER: MDS: 2.17.57.83 [ 269.488780] Lustre: DEBUG MARKER: OSS: 2.17.57.83 [ 270.776779] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 24 01:13:43 EDT 2026 [ 285.791058] Lustre: Unmounted lustre-client [ 321.201807] Key type lgssc unregistered [ 321.448650] LNet: 8401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.462432] LNetError: 8401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 321.491303] LNet: Removed LNI 192.168.203.24@tcp [ 322.377152] Key type .llcrypt unregistered [ 322.380601] Key type ._llcrypt unregistered [ 348.671760] Key type ._llcrypt registered [ 348.673531] Key type .llcrypt registered [ 348.893984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 348.903816] alg: No test for adler32 (adler32-zlib) [ 349.826878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 349.893144] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 349.922805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 355.151743] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 355.622120] LNet: 9292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.631636] LNetError: 9292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 355.649868] LNet: Removed LNI 192.168.203.24@tcp [ 356.711387] Key type .llcrypt unregistered [ 356.717774] Key type ._llcrypt unregistered [ 364.041054] Key type ._llcrypt registered [ 364.042701] Key type .llcrypt registered [ 365.583552] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 01:15:17 (1787548517) [ 365.968655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 365.988217] alg: No test for adler32 (adler32-zlib) [ 366.865090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.945859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.964907] LNet: 9707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.995352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.072136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 367.100060] LNet: 9714:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 373.690506] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 01:15:26 (1787548526) [ 373.969828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.993433] LNet: 10293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.008851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.061606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 380.616217] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 01:15:33 (1787548533) [ 380.943580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.961221] LNet: 10879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.994387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.072170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 386.865803] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 01:15:39 (1787548539) [ 387.363534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.391792] LNet: 11463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.412461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.539251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 394.385520] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 01:15:47 (1787548547) [ 394.472971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.490660] LNet: 12041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.530093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.621701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 395.013784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 401.205330] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 01:15:53 (1787548553) [ 401.288446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.356468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.467932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 401.982706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 402.323133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 408.713914] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 01:16:01 (1787548561) [ 408.965185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.991753] LNet: 13234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.001161] LNet: 13234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 409.037725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.135382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 416.091201] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 01:16:08 (1787548568) [ 416.205969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.242507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.352282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 416.385500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 416.874941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 416.918155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 417.377346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 417.431270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 417.958839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 418.021329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 418.431335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 418.483647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 419.022151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 419.064937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 419.554890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 419.600882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 420.102117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 420.144972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 420.470205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 420.528300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 420.907671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 420.935888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 421.398919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 421.446713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 421.899893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 421.956905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 422.404854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 422.463584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 429.050762] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 01:16:21 (1787548581) [ 429.292871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.328821] LNet: 14502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 429.334436] LNet: 14502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 429.359159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.467162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 429.516525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 435.580289] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 01:16:28 (1787548588) [ 435.937852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.973142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.070451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 436.093806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 442.699115] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 01:16:35 (1787548595) [ 442.799079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.853905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.955907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 443.004530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 449.851376] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 01:16:42 (1787548602) [ 450.038419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.103970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.200815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 450.238584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 457.156876] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 01:16:50 (1787548610) [ 457.370920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.409148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.476737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 457.507674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 463.273553] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 01:16:56 (1787548616) [ 463.383503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.400692] LNet: 17428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 463.405240] LNet: 17428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 463.436972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.550355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 463.597735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 469.997287] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 01:17:02 (1787548622) [ 470.158843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.227959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.323537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.777477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 477.575401] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 01:17:10 (1787548630) [ 477.698806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.764337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.085337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 478.320670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 485.274085] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 01:17:17 (1787548637) [ 485.369732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.418874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.580908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 485.857738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 492.075216] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 01:17:24 (1787548644) [ 492.194933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.261758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.945776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 493.236488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 499.611606] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 01:17:32 (1787548652) [ 499.739663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.795276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.917649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 499.969762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 507.470994] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 01:17:40 (1787548660) [ 507.732926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.786755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.870821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 507.914560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 515.587563] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 01:17:47 (1787548667) [ 515.790361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.849105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.953980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 522.165123] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 01:17:54 (1787548674) [ 522.249511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.315883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.412051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 528.508905] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 01:18:01 (1787548681) [ 528.644441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.679405] LNet: 22871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.693133] LNet: 22871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 528.738159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.811495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 529.232574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 535.653771] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 01:18:08 (1787548688) [ 535.852296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.896273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.991628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 536.058236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 542.396810] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:18:15 (1787548695) [ 542.538467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.581965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.678801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 542.709759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 549.840505] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:18:22 (1787548702) [ 550.103831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.148933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.209512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 550.251167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 558.874825] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:18:31 (1787548711) [ 559.238294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.324628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.466581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 559.594111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 567.424837] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:18:40 (1787548720) [ 567.611098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.692050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.819785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 567.893467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 567.926782] LNetError: 25819:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 568.003450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 568.096735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 574.818358] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:18:47 (1787548727) [ 575.008811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.076987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.213792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GIFdPtPmTW [ 575.295415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GIFdPtPmTW [ 575.372478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GIFdPtPmTW [ 582.235608] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:18:54 (1787548734) [ 582.336175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.402082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.501906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 582.533519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 582.581513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 589.115469] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 01:19:01 (1787548741) [ 589.222209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.268439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.328785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 589.354584] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 590.967728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 592.217641] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 594.088245] Key type lgssc registered [ 595.347363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 599.399726] LNetError: 27577:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.24@tcp99 has no available nets [ 602.165775] Key type lgssc unregistered [ 602.522366] LNetError: 28692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.552623] LNet: Removed LNI 192.168.203.24@tcp [ 603.276215] Key type .llcrypt unregistered [ 603.282780] Key type ._llcrypt unregistered [ 610.278381] Key type ._llcrypt registered [ 610.280789] Key type .llcrypt registered [ 611.577024] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:19:24 (1787548764) [ 612.067289] Key type .llcrypt unregistered [ 612.074409] Key type ._llcrypt unregistered [ 618.670359] Key type ._llcrypt registered [ 618.673207] Key type .llcrypt registered [ 618.831497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 618.845611] alg: No test for adler32 (adler32-zlib) [ 619.797324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.901571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 619.927314] LNet: 29613:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 619.933284] LNet: 29613:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 619.938878] LNet: 29613:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 619.958271] LNet: Message forwarding will be enabled in 2s [ 623.072168] LNet: Message forwarding enabled [ 623.500725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 623.524575] LNet: Message forwarding will stop in 2s to 6s [ 627.172770] LNet: Message forwarding disabled [ 628.110289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 628.144546] LNet: 29711:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 628.151599] LNet: 29711:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 628.157550] LNet: 29711:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 628.169761] LNet: Message forwarding will be enabled in 2s [ 628.372940] LNet: 29724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.910948] Key type .llcrypt unregistered [ 628.915427] Key type ._llcrypt unregistered [ 641.244534] Key type ._llcrypt registered [ 641.246724] Key type .llcrypt registered [ 642.450621] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:19:55 (1787548795) [ 642.973507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 642.989405] alg: No test for adler32 (adler32-zlib) [ 643.944499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.023866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 651.128631] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:20:03 (1787548803) [ 651.306424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.355800] LNet: 31130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.388334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.434283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 651.460528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 651.487509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 651.532509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 651.575750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 651.607492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 651.637865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 651.671805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 651.713481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 651.763413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 651.811550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 651.875155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 651.918613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 651.961488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 652.013553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 652.058725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 652.117087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 652.168631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 652.208638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 652.236132] LNetError: 31174:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 659.304934] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:20:11 (1787548811) [ 659.580299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.640754] LNet: 31745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.664724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.777317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 668.275715] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:20:20 (1787548820) [ 670.625638] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg324-server [ 672.712111] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:20:25 (1787548825) [ 672.862251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.889440] LNet: 32672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.955520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.095935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 673.115486] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 674.334658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[25-29]@tcp [ 674.362628] LNet: 32811:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 674.438293] LNetError: 32682:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.26@tcp is being used as a gateway but routing feature is not turned on [ 674.452829] LNetError: 32682:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.26@tcp has gone from up to down [ 683.176216] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:20:35 (1787548835) [ 685.747933] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg324-server [ 687.516314] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:20:40 (1787548840) [ 687.706762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.768335] LNet: 33743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.789281] LNetError: 33743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 687.848752] LNet: Removed LNI 192.168.203.24@tcp [ 687.915308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.963925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 688.012296] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 689.013831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[25-30/2]@tcp [ 689.042891] LNet: 33867:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 689.048732] LNet: 33867:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 689.069401] LNetError: 33753:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.25@tcp is being used as a gateway but routing feature is not turned on [ 689.076917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[25-30/2]@tcp [ 689.080226] LNetError: 33753:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 689.080256] LNetError: 33753:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.25@tcp has gone from up to down [ 689.080260] LNetError: 33753:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 695.550329] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:20:48 (1787548848) [ 695.673486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.711401] LNet: 34457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.729017] LNetError: 34457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.809403] LNet: Removed LNI 192.168.203.24@tcp [ 696.848620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.951493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 697.015406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.083948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.149124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.206265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.238996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.280887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 697.351488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.383753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.451097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.542382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.578558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.637121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 697.712765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.753820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.833920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.909143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.953321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.014917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 698.102995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.153734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.224972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.297803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.335741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.382801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 698.458223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.493205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.543741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.608486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.657178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.698714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 698.750558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.801426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.859564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 705.047764] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:20:57 (1787548857) [ 707.401279] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg324-server [ 709.536339] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:21:01 (1787548861) [ 711.574800] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg324-server [ 713.470099] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:21:06 (1787548866) [ 713.599415] LNet: 35854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.602191] LNet: 35854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 714.452644] Key type .llcrypt unregistered [ 714.459252] Key type ._llcrypt unregistered [ 720.773507] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 721.018411] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 721.021770] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 721.026430] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 723.635726] Key type ._llcrypt registered [ 723.637932] Key type .llcrypt registered [ 723.967787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 723.987183] alg: No test for adler32 (adler32-zlib) [ 725.040618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.133298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 725.163857] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 726.126689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 726.146097] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 729.303848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 729.335115] LNet: Removed LNI 10.1.2.3@tcp [ 729.517275] LNet: 36596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.526094] LNetError: 36596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.546926] LNet: Removed LNI 192.168.203.24@tcp [ 730.790870] Key type .llcrypt unregistered [ 730.795719] Key type ._llcrypt unregistered [ 748.623951] Key type ._llcrypt registered [ 748.627806] Key type .llcrypt registered [ 750.518136] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:21:42 (1787548902) [ 751.468306] Key type .llcrypt unregistered [ 751.470164] Key type ._llcrypt unregistered [ 759.394165] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 759.567085] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 759.581592] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 759.592881] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 762.162248] Key type ._llcrypt registered [ 762.164770] Key type .llcrypt registered [ 762.319372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 762.329460] alg: No test for adler32 (adler32-zlib) [ 763.215794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.263903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 763.286994] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 763.925375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 763.947935] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 767.161592] LNet: 38097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.174789] LNetError: 38097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 767.184643] LNet: Removed LNI 10.1.2.3@tcp [ 767.193699] LNet: Removed LNI 192.168.203.24@tcp [ 768.410522] Key type .llcrypt unregistered [ 768.438308] Key type ._llcrypt unregistered [ 784.644483] Key type ._llcrypt registered [ 784.646960] Key type .llcrypt registered [ 786.608645] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:22:18 (1787548938) [ 787.371459] Key type .llcrypt unregistered [ 787.373502] Key type ._llcrypt unregistered [ 797.346782] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 797.561438] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 797.564813] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 797.572366] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 800.265643] Key type ._llcrypt registered [ 800.273757] Key type .llcrypt registered [ 800.428269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 800.464711] alg: No test for adler32 (adler32-zlib) [ 801.461681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.588417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 801.643423] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 803.455283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 803.471296] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 804.781511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 804.794761] LNet: Removed LNI 10.1.2.3@tcp [ 804.804486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 804.827802] LNet: Removed LNI 10.1.2.31@tcp [ 804.850824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 804.881651] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 805.712492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 805.727807] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 806.635062] LNet: 40019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.640570] LNetError: 40019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 806.644460] LNet: Removed LNI 10.1.2.31@tcp [ 806.656863] LNet: Removed LNI 10.1.2.3@tcp [ 807.366455] Key type .llcrypt unregistered [ 807.372495] Key type ._llcrypt unregistered [ 817.624274] Key type ._llcrypt registered [ 817.626800] Key type .llcrypt registered [ 818.907590] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:22:51 (1787548971) [ 819.234584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 819.263249] alg: No test for adler32 (adler32-zlib) [ 820.164458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.212264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 820.229614] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 820.830094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.870225] LNet: 40975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.876625] LNetError: 40975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.897163] LNet: Removed LNI 192.168.203.24@tcp [ 820.921667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.991908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.24@tcp [ 821.018563] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 821.628156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.24@tcp [ 821.658189] LNet: Removed LNI 192.168.203.24@tcp [ 821.720890] LNet: 41110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.157550] Key type .llcrypt unregistered [ 822.160361] Key type ._llcrypt unregistered [ 832.231188] Key type ._llcrypt registered [ 832.232420] Key type .llcrypt registered [ 833.589598] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:23:06 (1787548986) [ 835.427036] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg324-server [ 837.014200] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:23:09 (1787548989) [ 837.770263] Key type .llcrypt unregistered [ 837.774626] Key type ._llcrypt unregistered [ 844.702992] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 844.976381] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 844.998554] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 845.009907] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 846.740151] Key type ._llcrypt registered [ 846.741531] Key type .llcrypt registered [ 846.994297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 847.019806] alg: No test for adler32 (adler32-zlib) [ 847.959434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.033475] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 848.917869] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 851.205073] LNet: 42945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.222339] LNetError: 42945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.236077] LNet: Removed LNI 192.168.203.24@tcp1 [ 851.296451] LNet: Removed LNI 10.1.2.3@tcp2 [ 852.210789] Key type .llcrypt unregistered [ 852.214102] Key type ._llcrypt unregistered [ 868.714683] Key type ._llcrypt registered [ 868.717662] Key type .llcrypt registered [ 871.075822] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:23:42 (1787549022) [ 871.621102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 871.653834] alg: No test for adler32 (adler32-zlib) [ 872.709660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.808956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 872.851243] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 873.721348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.771528] LNet: 43919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.789083] LNetError: 43919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.814621] LNet: Removed LNI 192.168.203.24@tcp [ 873.868192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.935336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 873.997650] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 881.714306] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:23:54 (1787549034) [ 881.986545] LNet: 44510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.999390] LNetError: 44510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.026891] LNet: Removed LNI 192.168.203.24@tcp [ 882.901932] Key type .llcrypt unregistered [ 882.905295] Key type ._llcrypt unregistered [ 890.328771] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 890.480539] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 890.484582] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 890.488075] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 893.086246] Key type ._llcrypt registered [ 893.088565] Key type .llcrypt registered [ 893.324936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 893.335061] alg: No test for adler32 (adler32-zlib) [ 894.276179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.346863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 894.379977] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 895.457757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 895.500887] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 898.393320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.433741] LNet: 45256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.441484] LNetError: 45256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 898.451743] LNet: Removed LNI 10.1.2.3@tcp [ 898.476554] LNet: Removed LNI 192.168.203.24@tcp [ 898.512914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.621363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 898.676834] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 898.692643] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 899.373921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.401568] LNet: 45283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.422038] LNetError: 45283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 899.447733] LNetError: 45283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 899.457033] LNet: Removed LNI 10.1.2.3@tcp [ 899.477713] LNet: Removed LNI 192.168.203.24@tcp [ 899.556068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.712559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 899.744072] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 899.747782] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 900.685701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 900.729088] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 902.354435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 902.379648] LNet: 45446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.389175] LNetError: 45446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 902.400478] LNetError: 45446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 902.413288] LNet: Removed LNI 10.1.2.3@tcp [ 902.415010] LNet: Removed LNI 192.168.203.24@tcp [ 902.438149] LNet: Removed LNI 192.168.203.24@tcp2 [ 902.520835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.685099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 902.742886] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 902.751615] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 902.759462] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 914.158977] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:24:25 (1787549065) [ 914.357919] LNet: 46040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.361403] LNetError: 46040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 914.380769] LNetError: 46040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 914.394063] LNet: Removed LNI 10.1.2.3@tcp [ 914.415719] LNet: Removed LNI 192.168.203.24@tcp [ 914.463366] LNet: Removed LNI 192.168.203.24@tcp2 [ 915.822084] Key type .llcrypt unregistered [ 915.824687] Key type ._llcrypt unregistered [ 923.428514] Key type ._llcrypt registered [ 923.430941] Key type .llcrypt registered [ 923.614832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 923.627644] alg: No test for adler32 (adler32-zlib) [ 924.647701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.744764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 924.809416] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 925.718603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.741985] LNet: 46446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.750630] LNetError: 46446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.765052] LNet: Removed LNI 192.168.203.24@tcp [ 925.795112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.874877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 925.906730] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 926.432514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.455868] LNet: 46476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.467521] LNetError: 46476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.485434] LNet: Removed LNI 192.168.203.24@tcp [ 926.523859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.603590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 926.633490] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 927.093907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.126229] LNet: Removed LNI 192.168.203.24@tcp [ 927.155085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.199363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 927.217678] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 927.779553] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 927.902843] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 927.907574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 927.916978] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 930.542444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.589062] LNet: 46584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.596735] LNet: 46584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 930.606418] LNetError: 46584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.612023] LNetError: 46584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 930.652366] LNet: Removed LNI 192.168.203.24@tcp [ 930.696974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.788548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 930.819147] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 937.470410] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:24:50 (1787549090) [ 937.571501] LNet: 47174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.578632] LNetError: 47174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.596194] LNet: Removed LNI 192.168.203.24@tcp [ 938.245795] Key type .llcrypt unregistered [ 938.248776] Key type ._llcrypt unregistered [ 943.817451] Key type ._llcrypt registered [ 943.825983] Key type .llcrypt registered [ 943.984564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 944.007444] alg: No test for adler32 (adler32-zlib) [ 944.885768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.928861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 944.976685] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 945.714662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.732969] LNet: 47583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.738795] LNetError: 47583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.762985] LNet: Removed LNI 192.168.203.24@tcp [ 945.829115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.928588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 945.970776] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 951.855605] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:25:04 (1787549104) [ 952.016878] LNet: 48171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.022914] LNetError: 48171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.042282] LNet: Removed LNI 192.168.203.24@tcp [ 952.572948] Key type .llcrypt unregistered [ 952.575852] Key type ._llcrypt unregistered [ 957.834295] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 957.960702] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 957.965354] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 957.975146] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 960.452604] Key type ._llcrypt registered [ 960.460419] Key type .llcrypt registered [ 960.639139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.650401] alg: No test for adler32 (adler32-zlib) [ 961.517652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.571637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 961.587097] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 962.295848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.315630] LNet: 48628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.318680] LNetError: 48628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.329066] LNet: Removed LNI 192.168.203.24@tcp [ 962.359847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.427759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 962.458525] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 968.220127] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:25:21 (1787549121) [ 968.378177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.418939] LNet: 49221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.431094] LNetError: 49221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.459925] LNet: Removed LNI 192.168.203.24@tcp [ 968.477752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.593935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 973.822367] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:25:26 (1787549126) [ 973.950327] LNet: 49806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.436619] Key type .llcrypt unregistered [ 974.438780] Key type ._llcrypt unregistered [ 980.235535] Key type ._llcrypt registered [ 980.242393] Key type .llcrypt registered [ 980.472778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 980.486756] alg: No test for adler32 (adler32-zlib) [ 981.406952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.468085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 981.524705] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 982.480349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.499654] LNet: 50209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.516398] LNetError: 50209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.540962] LNet: Removed LNI 192.168.203.24@tcp [ 982.568969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.662182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 982.695659] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 988.397223] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:25:41 (1787549141) [ 988.741705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.761568] LNet: 50803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.766486] LNetError: 50803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.782927] LNet: Removed LNI 192.168.203.24@tcp [ 988.818632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.895389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 994.232848] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:25:47 (1787549147) [ 995.534435] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 997.145522] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:25:50 (1787549150) [ 998.761876] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1000.192444] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:25:53 (1787549153) [ 1001.742800] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1003.814788] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:25:56 (1787549156) [ 1005.301255] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1007.186424] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:25:59 (1787549159) [ 1008.665741] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1010.104323] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:26:02 (1787549162) [ 1010.318568] LNet: 52855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.064304] Key type .llcrypt unregistered [ 1011.072276] Key type ._llcrypt unregistered [ 1016.753136] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1016.868578] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1016.881090] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1016.888786] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1018.346478] Key type ._llcrypt registered [ 1018.348353] Key type .llcrypt registered [ 1018.544615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1018.570580] alg: No test for adler32 (adler32-zlib) [ 1019.490944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1019.587472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1019.605557] LNet: Added LNI 192.168.203.24@tcp [8/512/0/180] [ 1019.613499] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1019.618736] LNet: 53172:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1019.686451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1019.697962] LNet: 53191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.702666] LNetError: 53191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1019.707050] LNet: Removed LNI 10.1.2.3@tcp [ 1019.709682] LNetError: 53191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.720037] LNet: Removed LNI 192.168.203.24@tcp [ 1019.743404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1019.827388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1019.855129] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1019.932778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1019.957797] LNet: 53216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.963656] LNetError: 53216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.977557] LNet: Removed LNI 192.168.203.24@tcp [ 1019.994649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.044232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1020.073962] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1020.081128] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1020.084108] LNet: 53224:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1020.193130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1020.210597] LNet: 53242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.218496] LNetError: 53242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1020.226249] LNet: Removed LNI 10.1.2.3@tcp [ 1020.230104] LNetError: 53242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.247738] LNet: Removed LNI 192.168.203.24@tcp [ 1020.264801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.321917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1020.351805] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1020.356524] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1020.358448] LNet: 53250:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1020.533435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1020.561534] LNet: 53276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.570359] LNetError: 53276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1020.578945] LNet: Removed LNI 10.1.2.3@tcp [ 1020.583989] LNetError: 53276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.603489] LNet: Removed LNI 192.168.203.24@tcp [ 1020.618732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.665046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1020.692841] LNet: Added LNI 192.168.203.24@tcp [8/256/0/182] [ 1020.699139] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1020.788959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1020.805220] LNet: 53304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.809705] LNetError: 53304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1020.814726] LNet: Removed LNI 10.1.2.3@tcp [ 1020.817293] LNetError: 53304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.825937] LNet: Removed LNI 192.168.203.24@tcp [ 1020.848786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.923747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1020.949633] LNet: Added LNI 192.168.203.24@tcp [8/256/0/182] [ 1020.954300] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1020.960770] LNet: 53312:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1021.061355] LNet: Removed LNI 10.1.2.3@tcp [ 1021.087618] LNet: Removed LNI 192.168.203.24@tcp [ 1021.725397] Key type .llcrypt unregistered [ 1021.728337] Key type ._llcrypt unregistered [ 1031.003542] Key type ._llcrypt registered [ 1031.010094] Key type .llcrypt registered [ 1032.203766] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 01:26:24 (1787549184) [ 1033.765750] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1035.246723] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:26:28 (1787549188) [ 1035.881751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1035.919568] alg: No test for adler32 (adler32-zlib) [ 1036.884326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.967748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1036.999068] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1037.795907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.24 [ 1037.824276] LNet: 54579:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1037.841311] LNet: Removed LNI 192.168.203.24@tcp [ 1044.254905] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:26:36 (1787549196) [ 1044.467401] LNet: 55142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.479415] LNetError: 55142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.545533] LNet: Removed LNI 192.168.203.24@tcp [ 1045.152416] Key type .llcrypt unregistered [ 1045.156889] Key type ._llcrypt unregistered [ 1053.060559] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1053.307853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1054.964088] Key type ._llcrypt registered [ 1054.967265] Key type .llcrypt registered [ 1055.201406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1055.233493] alg: No test for adler32 (adler32-zlib) [ 1056.757953] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1062.676831] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:26:55 (1787549215) [ 1062.770813] LNet: 56052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.777981] LNetError: 56052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1062.797869] LNet: Removed LNI 10.1.2.3@tcp [ 1063.421170] Key type .llcrypt unregistered [ 1063.423701] Key type ._llcrypt unregistered [ 1069.526600] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1069.732040] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1069.736995] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1069.745510] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1072.384435] Key type ._llcrypt registered [ 1072.387422] Key type .llcrypt registered [ 1072.519774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1072.531966] alg: No test for adler32 (adler32-zlib) [ 1073.492070] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1078.269852] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:27:11 (1787549231) [ 1079.562585] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1081.078952] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:27:13 (1787549233) [ 1081.198113] LNet: 57242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.206972] LNetError: 57242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1081.228131] LNet: Removed LNI 10.1.2.3@tcp [ 1081.813952] Key type .llcrypt unregistered [ 1081.816864] Key type ._llcrypt unregistered [ 1087.908674] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1088.112556] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1088.115478] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1088.126972] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1089.731959] Key type ._llcrypt registered [ 1089.734560] Key type .llcrypt registered [ 1089.905592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1089.922403] alg: No test for adler32 (adler32-zlib) [ 1090.878475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1090.970543] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1098.265756] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:27:30 (1787549250) [ 1098.733429] LNet: 58472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1098.740793] LNetError: 58472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1098.756820] LNet: Removed LNI 10.1.2.3@tcp [ 1099.347669] Key type .llcrypt unregistered [ 1099.352128] Key type ._llcrypt unregistered [ 1104.616764] Key type ._llcrypt registered [ 1104.620049] Key type .llcrypt registered [ 1104.792528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1104.801657] alg: No test for adler32 (adler32-zlib) [ 1105.699650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1105.731722] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1112.758777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1113.374889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_interrupt -r 1 [ 1113.449514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_interrupt -r 1 [ 1113.509761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1113.538097] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1113.777127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.812634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1113.858706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1113.900903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1113.952875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1113.968567] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1113.998309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.029225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.063583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.078574] LNet: 58913:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1114.094428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.119939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1114.151318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1114.186777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1114.234451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.274815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1114.328970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1114.899831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_dropped -r 1 [ 1114.946718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_dropped -r 1 [ 1114.975837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1114.999320] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1115.184635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1115.206723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1115.229422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1115.250317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1115.271337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1115.298855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1115.333712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1115.368585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1115.392227] LNet: 59022:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1115.408247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1115.443804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.484189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.521824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.564624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.602867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.646734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1116.205509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_aborted -r 1 [ 1116.253730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_aborted -r 1 [ 1116.295283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1116.315859] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1116.325987] LNet: Skipped 1 previous similar message [ 1116.528370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.553556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1116.585179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.622688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.672775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.702844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.735215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.775397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.801841] LNet: 59131:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1116.818521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.865896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.906981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.938515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.984837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1117.024630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1117.089515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1117.631771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_no_route -r 1 [ 1117.669272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_no_route -r 1 [ 1117.697179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1117.924299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1117.950458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1117.973858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1117.997236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1118.031570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1118.058611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.090789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.130589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.197845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1118.275921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.332514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1118.375158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.424349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.473993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1118.535313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1119.164506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_timeout -r 1 [ 1119.238931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_timeout -r 1 [ 1119.309622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1119.378158] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1119.400831] LNet: Skipped 1 previous similar message [ 1119.780687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.818962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1119.860166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.902167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.957993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.998810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1120.039299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1120.075094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1120.105468] LNet: 59350:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1120.115807] LNet: 59350:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1120.146595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.191873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1120.236163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.292596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.354481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.396198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.471480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1121.081902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_error -r 1 [ 1121.143345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_error -r 1 [ 1121.187464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1121.488787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.560744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1121.610575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.648309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.715900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1121.762488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1121.825176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1121.883376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1121.918504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1121.961740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.001665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.060269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.108329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.146338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.194127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1122.506414] LNet: 59504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.514649] LNetError: 59504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.564234] LNet: Removed LNI 192.168.203.24@tcp [ 1123.999823] Key type .llcrypt unregistered [ 1124.003079] Key type ._llcrypt unregistered [ 1144.970568] Key type ._llcrypt registered [ 1144.973239] Key type .llcrypt registered [ 1146.523177] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:28:18 (1787549298) [ 1147.813683] Key type .llcrypt unregistered [ 1147.817428] Key type ._llcrypt unregistered [ 1155.433418] Key type ._llcrypt registered [ 1155.435247] Key type .llcrypt registered [ 1155.715588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1155.746790] alg: No test for adler32 (adler32-zlib) [ 1156.614211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1156.634152] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1165.270257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1175.198991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1175.239716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1175.276858] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1176.737664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_interrupt -r 1 [ 1176.764935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_interrupt -r 1 [ 1176.804996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_interrupt -r 1 [ 1176.835559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_interrupt -r 1 [ 1176.858053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_interrupt -r 1 [ 1176.896628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_interrupt -r 1 [ 1176.931312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_interrupt -r 1 [ 1176.966262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_interrupt -r 1 [ 1177.003524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1177.026643] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1177.033667] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1177.244715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.268954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1177.290775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.308747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.336615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.365374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.412098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.458338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.486256] LNet: 60978:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1177.502786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.538122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.566580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.599536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.642879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.686521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.730613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1178.121961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_dropped -r 1 [ 1178.155348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_dropped -r 1 [ 1178.179776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_dropped -r 1 [ 1178.226474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_dropped -r 1 [ 1178.266632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_dropped -r 1 [ 1178.315831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_dropped -r 1 [ 1178.362337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_dropped -r 1 [ 1178.412395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_dropped -r 1 [ 1178.451283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1178.471421] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1178.477471] LNet: Skipped 2 previous similar messages [ 1178.687733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1178.722916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1178.763286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1178.793433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.827519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.856837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.881951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.906097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.919811] LNet: 61099:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1178.933506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1178.954519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.990218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1179.029212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.061745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.090539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1179.122699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1179.532214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_aborted -r 1 [ 1179.568951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_aborted -r 1 [ 1179.599579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_aborted -r 1 [ 1179.629946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_aborted -r 1 [ 1179.660645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_aborted -r 1 [ 1179.698602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_aborted -r 1 [ 1179.732550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_aborted -r 1 [ 1179.765649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_aborted -r 1 [ 1179.805433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1179.824945] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1179.831249] LNet: Skipped 2 previous similar messages [ 1180.000311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.028578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1180.057917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1180.107253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1180.149586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1180.184392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.209797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.238247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.252138] LNet: 61221:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1180.266446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.310545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.347159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.392975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.417904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.444447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.494033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1180.927299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_no_route -r 1 [ 1180.956441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_no_route -r 1 [ 1180.997686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_no_route -r 1 [ 1181.038131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_no_route -r 1 [ 1181.079532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_no_route -r 1 [ 1181.126024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_no_route -r 1 [ 1181.186099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_no_route -r 1 [ 1181.238695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_no_route -r 1 [ 1181.286290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1181.509477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.550446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.575990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.613752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.643572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.664896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.695563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.726776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.760390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.801642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.837107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.878689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.903787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.932717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.960218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1182.413433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_timeout -r 1 [ 1182.452347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_timeout -r 1 [ 1182.492686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_timeout -r 1 [ 1182.528358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_timeout -r 1 [ 1182.556576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_timeout -r 1 [ 1182.594346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_timeout -r 1 [ 1182.628953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_timeout -r 1 [ 1182.656860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_timeout -r 1 [ 1182.695766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1182.713825] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1182.717604] LNet: Skipped 5 previous similar messages [ 1182.904662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.929687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.967504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.006870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.037485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.070439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.108082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.153380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.176930] LNet: 61463:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.186810] LNet: 61463:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1183.205150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.237219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.265030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.306383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.334922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.369708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.398709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1183.833249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_error -r 1 [ 1183.857718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_error -r 1 [ 1183.885584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_error -r 1 [ 1183.926424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_error -r 1 [ 1183.955970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_error -r 1 [ 1183.981796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_error -r 1 [ 1184.014937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_error -r 1 [ 1184.046932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_error -r 1 [ 1184.079230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1184.272934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.293609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.321475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.348304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.380558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.411146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.438471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.465741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.496365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.527374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.557539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1184.596481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.644422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.678250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.708069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1188.871603] LNet: 61664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1188.877061] LNetError: 61664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1189.920439] LNet: Removed LNI 192.168.203.24@tcp [ 1189.922354] LNetError: 61664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1189.938848] LNet: Removed LNI 192.168.203.24@tcp1 [ 1190.287846] Key type .llcrypt unregistered [ 1190.290736] Key type ._llcrypt unregistered [ 1204.409124] Key type ._llcrypt registered [ 1204.412575] Key type .llcrypt registered [ 1205.409973] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:29:18 (1787549358) [ 1205.996201] Key type .llcrypt unregistered [ 1206.000503] Key type ._llcrypt unregistered [ 1210.787871] Key type ._llcrypt registered [ 1210.789306] Key type .llcrypt registered [ 1210.948210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1210.961571] alg: No test for adler32 (adler32-zlib) [ 1212.053336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1212.108857] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1220.599119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1221.389278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_dropped -r 1 [ 1221.415741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_dropped -r 1 [ 1221.466676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1221.512975] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1221.892136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1221.930064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1221.982869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1222.051252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1222.108358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1222.159553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1222.208585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1222.246985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1222.287860] LNet: 62917:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1222.331899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1222.405715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1222.445363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1222.472232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1222.507555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1222.537845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1222.594490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1223.264707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_error -r 1 [ 1223.312963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_error -r 1 [ 1223.356275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1223.382311] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1223.649590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1223.696074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1223.727642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1223.752219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1223.782225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1223.810831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1223.835801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1223.864463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1223.889066] LNet: 63026:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1223.919213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1224.096587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1224.157299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1224.187297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1224.233566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1224.287978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1224.336976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1225.128169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_timeout -r 1 [ 1225.184800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_timeout -r 1 [ 1225.265943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1225.301576] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1225.501938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1225.543671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1225.603369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1225.657087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1225.723989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1225.795414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1225.856156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1225.911130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1225.931996] LNet: 63136:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1225.954214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.008719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.041403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.068367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.107894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.161577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.244662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1226.771908] LNet: 63181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.786281] LNetError: 63181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.860974] LNet: Removed LNI 192.168.203.24@tcp [ 1227.425512] Key type .llcrypt unregistered [ 1227.431543] Key type ._llcrypt unregistered [ 1245.660649] Key type ._llcrypt registered [ 1245.664487] Key type .llcrypt registered [ 1246.952193] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:29:59 (1787549399) [ 1247.746866] Key type .llcrypt unregistered [ 1247.749293] Key type ._llcrypt unregistered [ 1253.564649] Key type ._llcrypt registered [ 1253.571967] Key type .llcrypt registered [ 1253.874093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1253.894067] alg: No test for adler32 (adler32-zlib) [ 1254.785347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1254.813049] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1262.280795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1275.488673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1275.553738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1275.597047] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1278.074933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_dropped -r 1 [ 1278.113803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e remote_dropped -r 1 [ 1278.157732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_dropped -r 1 [ 1278.190726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e remote_dropped -r 1 [ 1278.226868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e remote_dropped -r 1 [ 1278.275474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e remote_dropped -r 1 [ 1278.333283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e remote_dropped -r 1 [ 1278.397554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e remote_dropped -r 1 [ 1278.499825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1278.553572] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1278.572170] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.124@tcp [ 1278.946944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.979841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1279.020156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.111261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1279.158559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1279.221666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1279.289673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1279.385699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1279.418852] LNet: 64651:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1279.448629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1279.508605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1279.568536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1279.645541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1279.662600] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1279.674898] LNet: Skipped 3 previous similar messages [ 1279.697853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1279.744563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1279.809745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1279.871417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.919862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_error -r 1 [ 1280.994777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e remote_error -r 1 [ 1281.042582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_error -r 1 [ 1281.109478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e remote_error -r 1 [ 1281.158469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e remote_error -r 1 [ 1281.226163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e remote_error -r 1 [ 1281.278639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e remote_error -r 1 [ 1281.324299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e remote_error -r 1 [ 1281.356923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1281.377677] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1281.599351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1281.635150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1281.657601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1281.688635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1281.712038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1281.733996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1281.759816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1281.802635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1281.828212] LNet: 64774:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1281.848577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1281.893603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1281.955737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1282.009561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1282.048898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1282.082302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1282.113805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1282.146563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1282.639565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_timeout -r 1 [ 1282.674357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e remote_timeout -r 1 [ 1282.709983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_timeout -r 1 [ 1282.740320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e remote_timeout -r 1 [ 1282.770643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e remote_timeout -r 1 [ 1282.799757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e remote_timeout -r 1 [ 1282.821346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e remote_timeout -r 1 [ 1282.840574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e remote_timeout -r 1 [ 1282.860825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp [ 1283.042169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1283.087419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1283.127462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1283.158335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1283.212603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1283.253911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1283.298715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1283.364334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1283.424746] LNet: 64897:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1283.467488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1283.519580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1283.569428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1283.609486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1283.647737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1283.711569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1283.770065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1283.810906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.691745] LNet: 64979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.719478] LNetError: 64979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1290.730224] LNet: Removed LNI 192.168.203.24@tcp [ 1290.742482] LNet: Removed LNI 192.168.203.24@tcp1 [ 1291.423618] Key type .llcrypt unregistered [ 1291.427130] Key type ._llcrypt unregistered [ 1308.479632] Key type ._llcrypt registered [ 1308.481151] Key type .llcrypt registered [ 1309.462988] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:31:02 (1787549462) [ 1311.099614] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1312.751611] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:31:05 (1787549465) [ 1313.535304] Key type .llcrypt unregistered [ 1313.538086] Key type ._llcrypt unregistered [ 1320.990067] Key type ._llcrypt registered [ 1320.996406] Key type .llcrypt registered [ 1321.350602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1321.378504] alg: No test for adler32 (adler32-zlib) [ 1322.517209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1322.599905] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1332.876729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1333.716543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e network_timeout -r 1 [ 1333.767568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e network_timeout -r 1 [ 1333.814770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1333.869811] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1334.198942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1334.260622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1334.375570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1334.449248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1334.508517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1334.531868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1334.578519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1334.620247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1334.644226] LNet: 66528:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1334.675387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1334.733601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1334.760871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1334.816243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1334.877723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1334.890303] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1334.896863] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1334.930664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1335.321507] LNet: 66571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.351859] LNetError: 66571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1335.391437] LNet: Removed LNI 192.168.203.24@tcp [ 1336.039799] Key type .llcrypt unregistered [ 1336.042555] Key type ._llcrypt unregistered [ 1352.548958] Key type ._llcrypt registered [ 1352.553850] Key type .llcrypt registered [ 1352.729996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1352.747729] alg: No test for adler32 (adler32-zlib) [ 1353.630910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1353.653034] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1360.577870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1368.431805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1368.474559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1368.530711] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1369.694239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e network_timeout -r 1 [ 1369.745570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e network_timeout -r 1 [ 1369.790235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e network_timeout -r 1 [ 1369.839682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e network_timeout -r 1 [ 1369.891349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e network_timeout -r 1 [ 1369.923035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e network_timeout -r 1 [ 1369.975119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e network_timeout -r 1 [ 1370.024823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e network_timeout -r 1 [ 1370.083966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1370.107916] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1370.387701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1370.425226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1370.454478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1370.489752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1370.533769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1370.563720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1370.621249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1370.661084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1370.681452] LNet: 67452:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1370.697421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1370.738827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1370.788283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1370.827144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1370.898356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1370.943982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1375.048194] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1375.200101] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1375.467185] LNet: 67530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1375.473513] LNetError: 67530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1375.484027] LNet: Removed LNI 192.168.203.24@tcp [ 1375.494835] LNet: Removed LNI 192.168.203.24@tcp1 [ 1375.823590] Key type .llcrypt unregistered [ 1375.827654] Key type ._llcrypt unregistered [ 1391.777522] Key type ._llcrypt registered [ 1391.784141] Key type .llcrypt registered [ 1393.032238] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:32:25 (1787549545) [ 1393.402928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1393.430708] alg: No test for adler32 (adler32-zlib) [ 1394.368348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1394.464057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1394.505190] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1395.416856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1395.456838] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1396.366682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1396.438456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1396.502600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1396.561823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1396.642580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1396.726784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1396.767832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.24@tcp [ 1396.805839] LNetError: 68632:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.24@tcp to recovery [ 1397.472330] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1397.474957] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1399.520341] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1400.460562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1403.618159] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1404.061972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1404.135722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1404.207505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1404.255037] LNet: 68739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1404.260376] LNetError: 68739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1404.277627] LNet: Removed LNI 192.168.203.24@tcp [ 1404.292063] LNet: Removed LNI 192.168.203.24@tcp1 [ 1404.314767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1404.422059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1404.479882] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1405.237392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1405.265329] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1406.062155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1406.100808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1406.195956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1406.264801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1406.312578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1406.346777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1406.396585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1406.444738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.24@tcp [ 1406.474905] LNetError: 69012:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.24@tcp to recovery [ 1407.460040] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1410.147898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1413.601184] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1413.607489] LNet: Skipped 1 previous similar message [ 1418.204893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1418.266568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1418.305303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1424.461808] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:32:57 (1787549577) [ 1424.689744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1424.726253] LNet: 69728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1424.746946] LNetError: 69728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1424.752032] LNetError: 69728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1424.755893] LNet: Removed LNI 192.168.203.24@tcp [ 1424.778028] LNet: Removed LNI 192.168.203.24@tcp1 [ 1424.800194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1424.841201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1424.859126] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1425.549559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1425.571204] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1426.235714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1426.300817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1426.329598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1426.362537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1426.394895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1426.424115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1426.451456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.24@tcp --health 0 [ 1426.592077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1426.848799] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.24@tcp [ 1427.873668] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1427.884216] LNet: Skipped 1 previous similar message [ 1442.390407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1442.610623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1442.680611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1442.737395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.24@tcp --health 500 [ 1442.883791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1448.700509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1448.847150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1448.871389] LNet: 70212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1448.875936] LNetError: 70212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1448.882422] LNetError: 70212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1448.888388] LNet: Removed LNI 192.168.203.24@tcp [ 1448.910475] LNet: Removed LNI 192.168.203.24@tcp1 [ 1448.986543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.086505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1449.138839] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1450.161734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1450.179272] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1450.958026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1451.016363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1451.073286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1451.096672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1451.127146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1451.154761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1451.180250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1451.214777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.24@tcp --health 0 [ 1452.131506] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1452.140330] LNet: Skipped 3 previous similar messages [ 1452.653438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1462.918624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1462.940300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1462.961742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1468.744860] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:33:41 (1787549621) [ 1469.340279] LNet: 71258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.349560] LNetError: 71258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.354354] LNetError: 71258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1469.369059] LNet: Removed LNI 192.168.203.24@tcp [ 1469.382317] LNet: Removed LNI 192.168.203.24@tcp1 [ 1470.210216] Key type .llcrypt unregistered [ 1470.214397] Key type ._llcrypt unregistered [ 1477.676195] Key type ._llcrypt registered [ 1477.680897] Key type .llcrypt registered [ 1477.974534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1478.012616] alg: No test for adler32 (adler32-zlib) [ 1479.014617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1479.065613] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1487.323380] Lustre: DEBUG MARKER: Initial discovery [ 1487.565707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.124@tcp [ 1489.678591] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1489.896553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1489.940370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.124@tcp [ 1489.966556] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1489.992329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1490.041244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.124@tcp -r 1 -m GET -l 3 [ 1490.094173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.124@tcp -r 1 -m GET -e local_error [ 1490.141401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.124@tcp -r 1 -m PUT -l 6 [ 1490.336354] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1491.508230] Lustre: DEBUG MARKER: Force oleg324-client.virtnet to discover 192.168.203.124@tcp (in background) [ 1491.747772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.124@tcp [ 1493.059261] Lustre: DEBUG MARKER: Force oleg324-server to discover 192.168.203.24@tcp [ 1494.816433] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1495.247558] Lustre: DEBUG MARKER: Wait for 72025 [ 1496.588313] Lustre: DEBUG MARKER: Finished wait on 72025 [ 1496.728792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1496.778377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1496.780743] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1497.025468] LNet: 72477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1497.033579] LNetError: 72477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1497.059223] LNet: Removed LNI 192.168.203.24@tcp [ 1497.417725] Key type .llcrypt unregistered [ 1497.424709] Key type ._llcrypt unregistered [ 1509.823310] Key type ._llcrypt registered [ 1509.827082] Key type .llcrypt registered [ 1511.158780] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:34:23 (1787549663) [ 1511.633479] Key type .llcrypt unregistered [ 1511.636959] Key type ._llcrypt unregistered [ 1516.536679] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1516.738094] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1516.745401] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1516.755947] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1519.186990] Key type ._llcrypt registered [ 1519.190961] Key type .llcrypt registered [ 1519.375790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1519.387499] alg: No test for adler32 (adler32-zlib) [ 1520.304980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1520.410400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1520.465985] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1521.131630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1521.160979] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1529.417561] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:34:41 (1787549681) [ 1529.555101] LNet: 74549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.563252] LNetError: 74549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1529.578256] LNet: Removed LNI 10.1.2.3@tcp [ 1529.610898] LNet: Removed LNI 192.168.203.24@tcp [ 1530.688708] Key type .llcrypt unregistered [ 1530.697724] Key type ._llcrypt unregistered [ 1537.403700] Key type ._llcrypt registered [ 1537.407079] Key type .llcrypt registered [ 1537.598809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1537.617674] alg: No test for adler32 (adler32-zlib) [ 1538.544968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.595545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1538.613044] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1539.377491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1539.408446] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1540.492491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.24@tcp [ 1541.536318] LNetError: Unexpected error -22 connecting to 192.168.203.24@tcp at host 192.168.203.24:988 [ 1546.564860] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:34:59 (1787549699) [ 1546.812647] LNet: 75690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.822507] LNetError: 75690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1546.833450] LNet: Removed LNI 10.1.2.3@tcp [ 1546.854665] LNet: Removed LNI 192.168.203.24@tcp [ 1547.423423] Key type .llcrypt unregistered [ 1547.425267] Key type ._llcrypt unregistered [ 1553.322871] Key type ._llcrypt registered [ 1553.326634] Key type .llcrypt registered [ 1553.603137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1553.623750] alg: No test for adler32 (adler32-zlib) [ 1554.548542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.581415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1554.595677] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1555.134697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1555.152830] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 1555.937743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.24@tcp1 --nid 192.168.203.24@tcp2 [ 1559.908533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp1 192.168.203.24@tcp2 [ 1559.933935] LNetError: 76529:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.24@tcp1 to 192.168.203.24@tcp2 - Route table may be misconfigured [ 1559.967287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp2 192.168.203.24@tcp1 [ 1565.791580] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:35:18 (1787549718) [ 1566.056444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1566.090305] LNet: 77103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1566.099671] LNetError: 77103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1566.112257] LNet: Removed LNI 192.168.203.24@tcp1 [ 1567.143840] LNet: Removed LNI 192.168.203.24@tcp2 [ 1567.188108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.241746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1567.268514] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1567.916830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1567.952602] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1568.752549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 1568.800865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.24@tcp [ 1568.867907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp1 192.168.203.24@tcp1 [ 1568.912607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.24@tcp -d 192.168.203.24@tcp -e network_timeout [ 1568.949452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -e network_timeout [ 1568.980276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -e network_timeout [ 1569.017191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -e network_timeout [ 1569.062675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1569.100507] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1569.119982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1569.203447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1569.249128] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1569.311094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1575.017442] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:35:27 (1787549727) [ 1575.113583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.126650] LNet: 77956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.131589] LNetError: 77956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.136490] LNetError: 77956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1575.143541] LNet: Removed LNI 192.168.203.24@tcp [ 1576.169147] LNet: Removed LNI 192.168.203.24@tcp1 [ 1576.261156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.359308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1576.727742] LNet: 78006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.129592] Key type .llcrypt unregistered [ 1577.134483] Key type ._llcrypt unregistered [ 1590.349988] Key type ._llcrypt registered [ 1590.362300] Key type .llcrypt registered [ 1592.319611] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:35:44 (1787549744) [ 1592.869142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1592.884686] alg: No test for adler32 (adler32-zlib) [ 1593.981447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.295720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1598.438608] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:35:50 (1787549750) [ 1598.705307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.736769] LNet: 79105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.760970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.839691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1598.870309] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1599.738676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1599.779434] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1601.245673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1601.315723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp1 [ 1601.392259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp1 [ 1607.498512] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:36:00 (1787549760) [ 1608.900566] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg324-server [ 1610.497142] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:36:03 (1787549763) [ 1612.247408] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg324-server [ 1613.637987] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:36:06 (1787549766) [ 1616.075465] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg324-server [ 1617.670691] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:36:10 (1787549770) [ 1619.335265] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg324-server [ 1620.630990] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:36:13 (1787549773) [ 1622.152817] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg324-server [ 1623.500490] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:36:16 (1787549776) [ 1625.018971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg324-server [ 1626.356962] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:36:19 (1787549779) [ 1627.664612] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg324-server [ 1629.019464] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:36:22 (1787549782) [ 1630.443416] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg324-server [ 1631.601826] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:36:24 (1787549784) [ 1632.927499] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg324-server [ 1634.175962] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:36:27 (1787549787) [ 1634.286651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.308613] LNet: 82981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.321087] LNetError: 82981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.329347] LNet: Removed LNI 192.168.203.24@tcp [ 1635.375144] LNet: Removed LNI 192.168.203.24@tcp1 [ 1635.416224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.473326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1635.496317] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1636.175568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1636.231862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1636.417311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.442346] LNet: 83130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.450153] LNetError: 83130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1636.455554] LNetError: 83130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1636.470365] LNet: Removed LNI 192.168.203.24@tcp [ 1636.492451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.536153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1636.570112] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1637.259055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1637.434634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.453477] LNetError: 83275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1638.503584] LNet: Removed LNI 192.168.203.24@tcp [ 1638.543829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.587961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1638.602923] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1639.176089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1639.247306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1639.465962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.484833] LNet: 83422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.489897] LNet: 83422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1639.505883] LNetError: 83422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1639.522984] LNet: Removed LNI 192.168.203.24@tcp [ 1639.546312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.615277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1639.639154] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1640.362436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1640.630691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.704119] LNet: Removed LNI 192.168.203.24@tcp [ 1641.751451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.871504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1641.895835] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1642.712911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1642.810752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1643.095853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.204295] LNet: Removed LNI 192.168.203.24@tcp [ 1644.242214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.360484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1644.413265] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1645.158587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1645.378739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.413486] LNet: 83860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.430474] LNet: 83860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1645.439564] LNetError: 83860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1645.445503] LNetError: 83860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1645.461150] LNet: Removed LNI 192.168.203.24@tcp [ 1645.498958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.552752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1645.578720] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1646.491773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1646.581487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1646.866427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.910803] LNet: Removed LNI 192.168.203.24@tcp [ 1647.935154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.990905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1648.024413] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1648.612687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1648.906319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.960334] LNet: Removed LNI 192.168.203.24@tcp [ 1649.996508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.070443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1650.119494] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1650.976586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1651.082702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1651.425620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.518880] LNet: Removed LNI 192.168.203.24@tcp [ 1652.536208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.608157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1652.659668] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1653.481001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1653.712971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.750243] LNet: 84445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.761168] LNet: 84445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1653.770433] LNetError: 84445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1653.783182] LNetError: 84445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1654.826696] LNet: Removed LNI 192.168.203.24@tcp [ 1654.863057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.944128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1654.985243] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1655.737969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1655.811898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1656.014951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1657.067988] LNet: Removed LNI 192.168.203.24@tcp [ 1657.123879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.154864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1657.170325] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1657.790974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1659.407507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.459681] LNet: Removed LNI 192.168.203.24@tcp [ 1660.475980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.537573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1660.562128] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1661.553437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1661.645509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1661.693857] LNet: Removed LNI 192.168.203.24@tcp [ 1661.723564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.805642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1661.862173] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1662.713568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1662.801934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.838142] LNet: Removed LNI 192.168.203.24@tcp [ 1662.865945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.915540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1663.002760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1663.051661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.108625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1663.213669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1663.279610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.330978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1663.413520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1663.442966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.483189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1663.500985] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1664.250960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.280996] LNet: Removed LNI 192.168.203.24@tcp [ 1664.296721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.356234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1670.680536] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:37:03 (1787549823) [ 1670.760858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1670.781536] LNet: 85762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.793282] LNet: 85762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1670.814665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.895365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1670.913754] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/180] [ 1672.327439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1672.358332] LNetError: 85906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.369586] LNetError: 85906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1672.400493] LNet: Removed LNI 192.168.203.24@tcp231 [ 1672.430902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.536957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1672.576074] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/180] [ 1672.588651] LNet: 85914:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1673.069475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1673.105244] LNet: Removed LNI 192.168.203.24@tcp231 [ 1673.129114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1673.169370] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/180] [ 1674.358644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.406742] LNet: Removed LNI 192.168.203.24@tcp231 [ 1674.426710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.504140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1674.556035] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/0] [ 1674.571614] LNet: 86066:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1674.963707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1674.995821] LNet: Removed LNI 192.168.203.24@tcp231 [ 1675.015230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1675.043133] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/0] [ 1676.241839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.284656] LNet: Removed LNI 192.168.203.24@tcp231 [ 1676.312100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.381541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1676.417688] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/60] [ 1676.430815] LNet: 86218:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1676.891869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1676.933063] LNet: Removed LNI 192.168.203.24@tcp231 [ 1676.959087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1677.011874] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/60] [ 1678.103332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.125085] LNet: Removed LNI 192.168.203.24@tcp231 [ 1678.150273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.238353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1678.259191] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/180] [ 1678.762558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1678.803350] LNet: Removed LNI 192.168.203.24@tcp231 [ 1678.822888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1678.862679] LNet: Added LNI 192.168.203.24@tcp231 [8/256/0/180] [ 1687.355285] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:37:19 (1787549839) [ 1687.471783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.533016] LNet: Removed LNI 192.168.203.24@tcp231 [ 1687.557750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.607450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1687.624646] LNet: Added LNI 192.168.203.24@tcp232 [8/256/0/180] [ 1688.229939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1688.276989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1688.318164] LNet: Removed LNI 192.168.203.24@tcp232 [ 1688.342876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1688.380465] LNet: Added LNI 192.168.203.24@tcp232 [8/256/0/180] [ 1689.080600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1689.118292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1689.147438] LNet: Removed LNI 192.168.203.24@tcp232 [ 1689.166096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1689.216346] LNet: Added LNI 192.168.203.24@tcp232 [8/256/0/180] [ 1689.988903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1690.024206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1690.071354] LNet: Removed LNI 192.168.203.24@tcp232 [ 1690.090504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1690.126781] LNet: Added LNI 192.168.203.24@tcp232 [8/256/0/180] [ 1690.847060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1690.897171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1690.940783] LNet: Removed LNI 192.168.203.24@tcp232 [ 1690.968955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1697.264251] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:37:29 (1787549849) [ 1698.361797] Key type .llcrypt unregistered [ 1698.370450] Key type ._llcrypt unregistered [ 1704.154088] Key type ._llcrypt registered [ 1704.156019] Key type .llcrypt registered [ 1704.394800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1704.409755] alg: No test for adler32 (adler32-zlib) [ 1705.380536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1705.433607] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1712.858587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1723.614057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.697704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1723.730304] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1725.944728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_interrupt -r 2 [ 1726.007558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_interrupt -r 2 [ 1726.062404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_interrupt -r 2 [ 1726.104655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_interrupt -r 2 [ 1726.169818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_interrupt -r 2 [ 1726.222150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_interrupt -r 2 [ 1726.266863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_interrupt -r 2 [ 1726.341822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_interrupt -r 2 [ 1726.393882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp [ 1726.421452] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1726.436036] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1726.674732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1726.703273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1726.727849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1726.750414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1726.780893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1726.807889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1726.847425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1726.893828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1726.925827] LNet: 88873:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1726.943638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1727.002220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1727.046635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1727.105412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1727.144561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1727.173646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1727.685817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_dropped -r 2 [ 1727.732324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_dropped -r 2 [ 1727.773079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_dropped -r 2 [ 1727.841650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_dropped -r 2 [ 1727.902922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_dropped -r 2 [ 1727.953848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_dropped -r 2 [ 1727.999462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_dropped -r 2 [ 1728.055759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_dropped -r 2 [ 1728.107779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp [ 1728.140873] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1728.468756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1728.513845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1728.559388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1728.597611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1728.646457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1728.714778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1728.761440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1728.799607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1728.828292] LNet: 88993:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1728.864305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1728.921672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1728.974409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1729.044775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1729.108111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1729.140232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1729.742426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_aborted -r 2 [ 1729.777748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_aborted -r 2 [ 1729.815967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_aborted -r 2 [ 1729.868171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_aborted -r 2 [ 1729.923817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_aborted -r 2 [ 1729.991616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_aborted -r 2 [ 1730.049865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_aborted -r 2 [ 1730.116028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_aborted -r 2 [ 1730.214613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp [ 1730.251387] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1730.538184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1730.556198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1730.596620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1730.644704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1730.678864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1730.721859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1730.762923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1730.794611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1730.811917] LNet: 89113:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1730.825652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1730.854500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1730.898408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1730.940754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1730.992745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1731.036262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1731.402853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_no_route -r 2 [ 1731.432527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_no_route -r 2 [ 1731.473505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_no_route -r 2 [ 1731.513532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_no_route -r 2 [ 1731.537822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_no_route -r 2 [ 1731.577448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_no_route -r 2 [ 1731.647946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_no_route -r 2 [ 1731.701328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_no_route -r 2 [ 1731.752512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp [ 1731.992588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1732.029851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1732.059990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1732.114515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1732.156237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1732.203892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1732.256524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1732.322260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1732.395565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1732.446491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1732.474304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1732.502783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1732.543444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1732.574940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1733.186143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_timeout -r 2 [ 1733.250545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_timeout -r 2 [ 1733.341076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_timeout -r 2 [ 1733.404841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_timeout -r 2 [ 1733.435966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_timeout -r 2 [ 1733.475611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_timeout -r 2 [ 1733.555853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_timeout -r 2 [ 1733.611116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_timeout -r 2 [ 1733.666453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp [ 1733.713782] LNet: There was an unexpected network error while writing to 192.168.203.124: rc = -22 [ 1733.737375] LNet: Skipped 1 previous similar message [ 1734.053946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1734.091487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1734.123811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1734.163425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1734.189333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1734.222452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1734.248482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1734.274404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1734.288280] LNet: 89354:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1734.293652] LNet: 89354:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1734.311820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1734.346891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1734.373257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1734.403611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1734.427097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1734.459392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1734.951972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_dropped -r 2 [ 1734.993183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e remote_dropped -r 2 [ 1735.019674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_dropped -r 2 [ 1735.067075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e remote_dropped -r 2 [ 1735.121578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e remote_dropped -r 2 [ 1735.164716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e remote_dropped -r 2 [ 1735.235900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e remote_dropped -r 2 [ 1735.312933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e remote_dropped -r 2 [ 1735.362438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp [ 1735.411243] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.124@tcp [ 1735.676812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1735.704992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1735.728040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1735.753406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1735.787861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1735.816798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1735.844229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1735.870572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1735.893786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1735.920804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1735.948453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1735.987972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1736.018215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1736.070169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1741.576371] LNet: 89552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.586966] LNetError: 89552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.594288] LNet: Removed LNI 192.168.203.24@tcp [ 1741.603642] LNet: Removed LNI 192.168.203.24@tcp1 [ 1742.036649] Key type .llcrypt unregistered [ 1742.039605] Key type ._llcrypt unregistered [ 1761.779919] Key type ._llcrypt registered [ 1761.781871] Key type .llcrypt registered [ 1763.348377] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 01:38:35 (1787549915) [ 1763.778608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1763.791216] alg: No test for adler32 (adler32-zlib) [ 1764.803108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.886344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1764.950451] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1765.939262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1765.971921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.24@tcp [ 1766.001511] LNetError: 90532:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.24@tcp to recovery [ 1766.079685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.24@tcp [ 1766.173641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1773.136239] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 01:38:45 (1787549925) [ 1773.229708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.256900] LNet: 91115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1773.266684] LNetError: 91115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1773.298437] LNet: Removed LNI 192.168.203.24@tcp [ 1773.325836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.394061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1773.442459] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1774.513822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1774.532815] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 1775.634991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.24@tcp [ 1775.664674] LNetError: 91388:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.24@tcp to recovery [ 1776.416467] LNet: 1 local NIs in recovery (showing 1): 192.168.203.24@tcp [ 1776.422823] LNet: There was an unexpected network error while writing to 192.168.203.24: rc = -22 [ 1776.881762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.24@tcp [ 1786.219549] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 01:38:58 (1787549938) [ 1788.281380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg324-server [ 1789.928504] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 01:39:02 (1787549942) [ 1792.086258] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg324-server [ 1794.288839] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 01:39:06 (1787549946) [ 1796.935899] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg324-server [ 1799.462625] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:39:11 (1787549951) [ 1799.541375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.564620] LNet: 93012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.576555] LNetError: 93012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.579682] LNet: Removed LNI 192.168.203.24@tcp [ 1799.589682] LNet: Removed LNI 192.168.203.24@tcp1 [ 1799.608839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.682414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1799.704976] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1800.619866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1800.731832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1800.818289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1801.044232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1801.120842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1801.329783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1810.366961] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:39:22 (1787549962) [ 1810.532541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1810.590706] LNet: 93745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1810.618532] LNetError: 93745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1810.631657] LNetError: 93745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1810.701254] LNet: Removed LNI 192.168.203.24@tcp [ 1810.790263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1810.895809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1810.940218] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1820.886981] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:39:33 (1787549973) [ 1821.448921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.523368] LNet: 94455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.532943] LNetError: 94455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.547397] LNet: Removed LNI 192.168.203.24@tcp [ 1821.572406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.665383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.724655] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1831.613500] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:39:43 (1787549983) [ 1833.719518] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1835.702659] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:39:48 (1787549988) [ 1836.824841] LNet: 95491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.830438] LNetError: 95491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1836.842214] LNet: Removed LNI 192.168.203.24@tcp [ 1838.064909] Key type .llcrypt unregistered [ 1838.068412] Key type ._llcrypt unregistered [ 1844.716971] Key type ._llcrypt registered [ 1844.719569] Key type .llcrypt registered [ 1844.922748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1844.941757] alg: No test for adler32 (adler32-zlib) [ 1845.887761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1845.956098] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1854.075909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 1860.964519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.124@tcp [ 1861.549923] LNet: 95916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1861.562957] LNetError: 95916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1861.610895] LNet: Removed LNI 192.168.203.24@tcp [ 1862.107254] Key type .llcrypt unregistered [ 1862.109248] Key type ._llcrypt unregistered [ 1876.114296] Key type ._llcrypt registered [ 1876.115747] Key type .llcrypt registered [ 1877.361476] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1879.271339] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1881.543244] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:40:33 (1787550033) [ 1883.689485] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1885.677860] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:40:37 (1787550037) [ 1888.564458] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg324-server [ 1891.157449] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:40:42 (1787550042) [ 1893.724784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg324-server [ 1896.140809] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:40:48 (1787550048) [ 1897.549489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1897.679638] alg: No test for adler32 (adler32-zlib) [ 1898.732661] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1898.951824] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1900.664349] Key type lgssc registered [ 1901.747686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1919.311981] Key type lgssc unregistered [ 1919.563287] LNet: 99105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.568867] LNetError: 99105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.583483] LNet: Removed LNI 192.168.203.24@tcp [ 1920.197097] Key type .llcrypt unregistered [ 1920.198872] Key type ._llcrypt unregistered [ 1931.890979] Key type ._llcrypt registered [ 1931.892584] Key type .llcrypt registered [ 1933.105041] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 01:41:25 (1787550085) [ 1933.655953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1933.678951] alg: No test for adler32 (adler32-zlib) [ 1934.678133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1934.781908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1934.842678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1934.915616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1934.970888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1941.394368] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 01:41:34 (1787550094) [ 1941.507787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1941.555556] LNet: 100512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.604861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.725219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1941.758337] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1942.577970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1942.604249] LNet: 100663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1942.610226] LNetError: 100663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1942.630536] LNet: Removed LNI 192.168.203.24@tcp [ 1942.650575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1942.702619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1942.736492] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1943.254358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.277403] LNetError: 100692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.311255] LNet: Removed LNI 192.168.203.24@tcp [ 1943.330334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.393233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1943.420532] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1949.916123] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 01:41:42 (1787550102) [ 1950.074255] LNet: 101278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1950.086839] LNet: 101278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1950.096437] LNetError: 101278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1950.116341] LNet: Removed LNI 192.168.203.24@tcp [ 1950.919865] Key type .llcrypt unregistered [ 1950.923635] Key type ._llcrypt unregistered [ 1956.866968] Key type ._llcrypt registered [ 1956.872504] Key type .llcrypt registered [ 1957.107737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1957.125372] alg: No test for adler32 (adler32-zlib) [ 1958.046890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.138094] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 1958.143884] LNet: *** cfs_fail_loc=e003, val=100*** [ 1958.807351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 1958.830875] LNet: *** cfs_fail_loc=e003, val=100*** [ 1958.837249] LNet: Skipped 2 previous similar messages [ 1958.845421] Ping reply received with mapping: [ 1958.848922] found metadata reply NI[192.168.203.24@tcp] [ 1958.858824] 18 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1958.863200] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1958.868025] 5a 5a 5a 5a 5a 5a 5a 5a [ 1959.000438] LNet: 101683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1959.020308] LNetError: 101683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1959.041814] LNet: Removed LNI 192.168.203.24@tcp [ 1959.803406] Key type .llcrypt unregistered [ 1959.807801] Key type ._llcrypt unregistered [ 1971.086517] Key type ._llcrypt registered [ 1971.089670] Key type .llcrypt registered [ 1972.200489] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 01:42:04 (1787550124) [ 1972.732636] Key type .llcrypt unregistered [ 1972.734419] Key type ._llcrypt unregistered [ 1978.798095] Key type ._llcrypt registered [ 1978.802024] Key type .llcrypt registered [ 1978.971887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1979.004525] alg: No test for adler32 (adler32-zlib) [ 1980.145181] LNetError: 102740:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1980.460578] Key type .llcrypt unregistered [ 1980.463141] Key type ._llcrypt unregistered [ 1984.902634] Key type ._llcrypt registered [ 1984.905418] Key type .llcrypt registered [ 1986.078096] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 01:42:18 (1787550138) [ 1986.821277] Key type .llcrypt unregistered [ 1986.824450] Key type ._llcrypt unregistered [ 1993.384120] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1993.389374] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1995.917208] Key type ._llcrypt registered [ 1995.921750] Key type .llcrypt registered [ 1996.151047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1996.171842] alg: No test for adler32 (adler32-zlib) [ 1997.141965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1997.226651] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2005.778107] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 01:42:38 (1787550158) [ 2005.947142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2005.981729] LNet: 104539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.991823] LNetError: 104539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2006.010374] LNet: Removed LNI 10.1.2.3@tcp [ 2006.021349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.776581] Key type .llcrypt unregistered [ 2006.781844] Key type ._llcrypt unregistered [ 2013.017712] Key type ._llcrypt registered [ 2013.023374] Key type .llcrypt registered [ 2013.185951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2013.200575] alg: No test for adler32 (adler32-zlib) [ 2014.160448] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2014.229658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 2014.395412] LNet: 104835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2014.406974] LNetError: 104835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2015.463645] LNet: Removed LNI 192.168.203.24@tcp [ 2016.165695] Key type .llcrypt unregistered [ 2016.172060] Key type ._llcrypt unregistered [ 2027.287644] Key type ._llcrypt registered [ 2027.292184] Key type .llcrypt registered [ 2028.433831] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 01:43:01 (1787550181) [ 2029.009355] Key type .llcrypt unregistered [ 2029.015288] Key type ._llcrypt unregistered [ 2034.910844] Key type ._llcrypt registered [ 2034.914360] Key type .llcrypt registered [ 2035.069334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2035.079879] alg: No test for adler32 (adler32-zlib) [ 2041.392166] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 01:43:14 (1787550194) [ 2041.531733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2041.585785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2041.642479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2041.665091] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2042.279671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2042.321979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2042.348819] LNet: Removed LNI 192.168.203.24@tcp [ 2042.656508] LNet: 106591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2043.056587] Key type .llcrypt unregistered [ 2043.060426] Key type ._llcrypt unregistered [ 2053.058167] Key type ._llcrypt registered [ 2053.062149] Key type .llcrypt registered [ 2054.491896] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 01:43:26 (1787550206) [ 2054.884076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2054.902861] alg: No test for adler32 (adler32-zlib) [ 2055.833712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2055.908771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2055.952025] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2056.584736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 2056.622376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.24@tcp [ 2056.645552] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2056.653856] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2056.661326] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2056, 0, 0 [ 2056.673831] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2056.681718] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2056.687213] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2056.692941] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2056, 0, 0 [ 2056.701273] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2056.706675] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2056.711559] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2056.715757] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2056, 0, 2106 [ 2056.721533] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2056.724845] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2056.729831] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2056.734463] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2056, 0, 2106 [ 2056.740396] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2063.222506] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 01:43:35 (1787550215) [ 2063.754613] LNet: 108130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.768166] LNetError: 108130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2063.801130] LNet: Removed LNI 192.168.203.24@tcp [ 2064.759621] Key type .llcrypt unregistered [ 2064.762648] Key type ._llcrypt unregistered [ 2072.502119] Key type ._llcrypt registered [ 2072.514977] Key type .llcrypt registered [ 2072.873634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2072.903504] alg: No test for adler32 (adler32-zlib) [ 2073.907956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2073.953321] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2082.827755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 2093.201732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2093.299767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2093.368833] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 2094.474669] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2094.720162] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2094.734395] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2094.741331] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2097.357186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2097.394496] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2111.500706] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 01:44:23 (1787550263) [ 2111.828316] LNet: 109641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2111.836114] LNetError: 109641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2111.845650] LNet: Removed LNI 192.168.203.24@tcp [ 2112.864540] LNet: Removed LNI 192.168.203.24@tcp1 [ 2112.870148] LNetError: 109641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2112.889109] LNetError: 109641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2112.920468] LNet: Removed LNI 10.1.2.3@tcp99 [ 2114.135076] Key type .llcrypt unregistered [ 2114.141172] Key type ._llcrypt unregistered [ 2122.471885] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2122.475043] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2125.142897] Key type ._llcrypt registered [ 2125.145624] Key type .llcrypt registered [ 2125.397475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2125.418078] alg: No test for adler32 (adler32-zlib) [ 2126.422919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.506109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2126.552077] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2127.558381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2127.593777] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2130.684584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2130.757830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2130.831378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2130.857586] LNetError: 110391:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2130.928720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2131.003766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2131.462979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2131.564912] LNet: 110411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.577314] LNetError: 110411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2131.591690] LNet: Removed LNI 10.1.2.3@tcp [ 2132.649158] LNet: Removed LNI 192.168.203.24@tcp [ 2133.673851] Key type .llcrypt unregistered [ 2133.679632] Key type ._llcrypt unregistered [ 2148.863208] Key type ._llcrypt registered [ 2148.865950] Key type .llcrypt registered [ 2150.723157] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 01:45:02 (1787550302) [ 2151.134059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2151.164660] alg: No test for adler32 (adler32-zlib) [ 2152.126517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2152.228592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2152.252887] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2153.187941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 2153.286252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg324-client@tcp [ 2160.459265] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 01:45:12 (1787550312) [ 2160.563290] LNet: 111940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2160.577816] LNetError: 111940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2160.616545] LNet: Removed LNI 192.168.203.24@tcp [ 2161.322465] Key type .llcrypt unregistered [ 2161.326952] Key type ._llcrypt unregistered [ 2168.275486] Key type ._llcrypt registered [ 2168.282471] Key type .llcrypt registered [ 2168.690793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2168.748745] alg: No test for adler32 (adler32-zlib) [ 2169.620306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2169.636813] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2169.672893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2175.375405] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 01:45:28 (1787550328) [ 2176.789819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2178.448277] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 01:45:31 (1787550331) [ 2180.026024] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2181.774756] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 01:45:34 (1787550334) [ 2182.099823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2182.116933] LNet: 113376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.122278] LNetError: 113376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2182.132364] LNet: Removed LNI 192.168.203.24@tcp [ 2182.147423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.202412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2182.219557] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2182.867561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2182.906393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2188.800387] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 01:45:41 (1787550341) [ 2188.918783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2188.941182] LNet: 114083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2188.952470] LNetError: 114083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2188.989034] LNet: Removed LNI 192.168.203.24@tcp [ 2189.011673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.052041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2189.080636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2189.670328] Key type .llcrypt unregistered [ 2189.674020] Key type ._llcrypt unregistered [ 2199.810497] Key type ._llcrypt registered [ 2199.813258] Key type .llcrypt registered [ 2201.368959] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 01:45:53 (1787550353) [ 2201.790833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2201.801370] alg: No test for adler32 (adler32-zlib) [ 2202.792876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2202.892068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2202.929125] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2203.799842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2203.889091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2210.131176] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 01:46:03 (1787550363) [ 2210.399335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2210.457131] LNet: 115643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2210.473275] LNetError: 115643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2210.495324] LNet: Removed LNI 192.168.203.24@tcp [ 2210.525429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.580876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2210.617487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2217.208399] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 01:46:09 (1787550369) [ 2217.330179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2217.382272] LNet: 116218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.420438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.528182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2217.777472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2217.800809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2217.866559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2218.329995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2218.382788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2218.475086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2218.511393] LNet: 116258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.551229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.623939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2218.866552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2218.904037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2218.946307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2219.214555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2219.237144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2225.643318] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 01:46:18 (1787550378) [ 2225.841701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2225.853654] LNet: 116858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2225.868828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.921721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2225.942030] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2226.703564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2226.742643] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 2228.233580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2228.409084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2228.912278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2228.963144] LNetError: 117138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2228.971490] LNet: Removed LNI 192.168.203.24@tcp [ 2228.995721] LNet: Removed LNI 192.168.203.24@tcp2 [ 2229.085792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2229.226506] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2229.243948] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 2229.256295] LNet: 117144:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2230.275444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2230.712738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2230.747435] LNet: 117181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2230.754953] LNet: 117181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2230.767444] LNetError: 117181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2230.776654] LNetError: 117181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2230.804502] LNet: Removed LNI 192.168.203.24@tcp [ 2230.813066] LNet: Removed LNI 192.168.203.24@tcp2 [ 2230.830782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2230.876136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2230.925521] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2232.525889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2232.590249] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 2234.087321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2234.207221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2234.320701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2234.694493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.719298] LNetError: 117465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2234.722800] LNetError: 117465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2235.747371] LNet: Removed LNI 192.168.203.24@tcp [ 2236.848041] LNet: Removed LNI 192.168.203.24@tcp2 [ 2236.998419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2237.113961] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2237.118796] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 2237.125862] LNet: 117472:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2237.484384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2237.650572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2237.674563] LNet: Removed LNI 192.168.203.24@tcp [ 2238.699205] LNet: Removed LNI 192.168.203.24@tcp2 [ 2238.739174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2238.840785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2238.891657] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2239.981393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2240.006938] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 2241.452451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2241.636064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2242.134597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2242.189727] LNet: 117788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.197363] LNet: 117788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2242.230545] LNetError: 117788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2242.255850] LNetError: 117788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2242.267043] LNet: Removed LNI 192.168.203.24@tcp [ 2242.283503] LNet: Removed LNI 192.168.203.24@tcp2 [ 2242.313705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2242.384548] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2242.389661] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 2242.394985] LNet: 117794:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2242.736874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2242.915160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2244.005183] LNet: Removed LNI 192.168.203.24@tcp [ 2244.040129] LNet: Removed LNI 192.168.203.24@tcp2 [ 2244.094321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.201376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2244.263599] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2245.622888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2245.695891] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 2247.525225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2247.591718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2247.679200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2248.099812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2249.185796] LNet: Removed LNI 192.168.203.24@tcp [ 2250.227585] LNet: Removed LNI 192.168.203.24@tcp2 [ 2250.347195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2250.499118] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2250.525985] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 2250.538276] LNet: 118123:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2251.643395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2251.839327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2251.873787] LNetError: 118160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.882800] LNetError: 118160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2251.925394] LNet: Removed LNI 192.168.203.24@tcp [ 2253.006050] LNet: Removed LNI 192.168.203.24@tcp2 [ 2253.050776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2253.144288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2253.215578] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2254.392540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2254.437436] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 2256.198553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2256.235274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2256.326856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2256.990880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2258.087203] LNet: Removed LNI 192.168.203.24@tcp [ 2258.099199] LNet: Removed LNI 192.168.203.24@tcp2 [ 2258.134390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2258.221191] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2258.227832] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 2258.233684] LNet: 118463:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2258.865550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2259.545694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2259.586941] LNet: 118515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2259.591949] LNet: 118515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2260.642561] LNet: Removed LNI 192.168.203.24@tcp [ 2260.665452] LNet: Removed LNI 192.168.203.24@tcp2 [ 2260.691434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2260.764827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2260.823435] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2261.795239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2261.823961] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 2263.354104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2263.435748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2263.488928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2263.561565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2264.105181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2265.225407] LNet: Removed LNI 192.168.203.24@tcp [ 2266.282563] LNet: Removed LNI 192.168.203.24@tcp2 [ 2266.346541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2266.466221] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2266.471426] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 2266.475845] LNet: 118823:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2266.982594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2274.778113] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2276.392377] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 01:47:09 (1787550429) [ 2276.589770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2276.615126] LNetError: 119582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.621109] LNetError: 119582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2276.634860] LNet: Removed LNI 192.168.203.24@tcp [ 2277.674311] LNet: Removed LNI 192.168.203.24@tcp2 [ 2277.755954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2278.216175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2278.240741] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2279.026447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2279.042691] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2280.001757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2280.045624] LNet: Removed LNI 10.1.2.3@tcp [ 2280.066691] LNet: Removed LNI 192.168.203.24@tcp [ 2280.089561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2280.184123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2280.262914] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2280.270442] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2280.871558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2280.899811] LNet: Removed LNI 192.168.203.24@tcp [ 2280.910655] LNet: Removed LNI 10.1.2.3@tcp [ 2287.470393] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 01:47:20 (1787550440) [ 2287.615242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2287.689440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2287.743895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2287.764185] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2288.525784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2288.619850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 2288.685347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 2288.722500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 2288.767182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 2288.827336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 2288.901698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 2288.974788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 2289.043182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 2289.099266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 2289.180384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp [ 2289.433439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2297.321657] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 01:47:29 (1787550449) [ 2297.478504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2297.535453] LNet: 121192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2297.554053] LNet: 121192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2298.600961] LNet: Removed LNI 192.168.203.24@tcp [ 2298.672344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2298.905560] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2299.015485] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2299.019615] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2299.023602] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2300.369812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2300.427177] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2301.248853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2301.269258] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2303.263851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2303.332986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2303.422417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.24@tcp -m PUT -e local_timeout -r 1 [ 2303.525401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2303.594520] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2303.601951] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2303.969898] LNetError: 121255:0:(lib-move.c:4801:lnet_parse()) 192.168.203.24@tcp, src 192.168.203.24@tcp: Dropping PUT (error -108 looking up sender) [ 2304.997571] LNet: Removed LNI 10.1.2.3@tcp [ 2306.023052] LNet: Removed LNI 192.168.203.24@tcp [ 2306.472968] Key type .llcrypt unregistered [ 2306.475219] Key type ._llcrypt unregistered [ 2320.418958] Key type ._llcrypt registered [ 2320.421694] Key type .llcrypt registered [ 2322.498807] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 01:47:54 (1787550474) [ 2323.000743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2323.013571] alg: No test for adler32 (adler32-zlib) [ 2323.930039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2324.236811] LNet: 122570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.901455] Key type .llcrypt unregistered [ 2324.906740] Key type ._llcrypt unregistered [ 2330.903597] Key type ._llcrypt registered [ 2330.908736] Key type .llcrypt registered [ 2331.169238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2331.195291] alg: No test for adler32 (adler32-zlib) [ 2332.187267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2332.234481] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2340.132757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 2340.296045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2340.776209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.124@tcp [ 2341.095859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2341.162353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2341.209634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2341.252448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2341.304246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2341.351589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2341.400842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2341.449856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2341.475382] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.124@tcp [ 2341.482907] LNet: 123025:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2341.512885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2341.561239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2341.618404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2341.648685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2341.679156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2341.703881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2342.422816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2342.678330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2342.710914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2342.774742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2342.822487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2342.872334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2342.909032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2342.969919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2343.033973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2343.060398] LNet: 123142:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2343.101326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2343.144961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2343.220976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2343.253585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2343.288769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2343.335790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2344.020198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.124@tcp [ 2344.231134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2344.263463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2344.303613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2344.329628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2344.357935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2344.385108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2344.438564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2344.472753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2344.493829] LNet: 123259:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2344.513753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2344.553529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2344.578664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2344.610508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2344.637656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2344.674605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2345.266642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2345.495803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2345.534299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2345.563954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2345.590965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2345.615410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2345.640927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2345.675211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2345.708596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2345.742179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2345.769901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2345.800967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2345.848556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2345.886905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2345.920184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2346.324826] LNet: 123426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2346.335821] LNetError: 123426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2346.360209] LNet: Removed LNI 192.168.203.24@tcp [ 2346.791447] Key type .llcrypt unregistered [ 2346.799725] Key type ._llcrypt unregistered [ 2365.181551] Key type ._llcrypt registered [ 2365.183648] Key type .llcrypt registered [ 2366.639454] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 01:48:39 (1787550519) [ 2367.272429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2367.308577] alg: No test for adler32 (adler32-zlib) [ 2368.240893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2368.553856] LNet: 124315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2369.113550] Key type .llcrypt unregistered [ 2369.119587] Key type ._llcrypt unregistered [ 2376.481579] Key type ._llcrypt registered [ 2376.493722] Key type .llcrypt registered [ 2376.783732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2376.802988] alg: No test for adler32 (adler32-zlib) [ 2377.815333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2377.840308] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2386.944414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp [ 2397.231660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2397.336578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2397.392468] LNet: Added LNI 192.168.203.24@tcp1 [8/256/0/180] [ 2398.928667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2400.028603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.124@tcp [ 2400.416951] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.124@tcp [ 2400.422829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2400.536155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2400.603643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2400.639876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2400.708489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2400.782646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2400.829348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2400.900836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2400.916489] LNet: 124973:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2400.931464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2400.958963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2400.982924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2401.016891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2401.046696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2401.077333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2401.755421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2402.165977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2402.208170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2402.245709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2402.281624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2402.344506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2402.403566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2402.461573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2402.529200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2402.556437] LNet: 125090:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2402.595481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2402.646765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2402.738054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2402.780875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2402.835903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2402.884945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2403.880381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.124@tcp [ 2404.246981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2404.346790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2404.413907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2404.450583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2404.470577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2404.492510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2404.516203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2404.624389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2404.643230] LNet: 125208:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2404.652103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2404.672264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2404.695682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2404.730158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2404.751611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2404.775692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2405.773228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2406.213306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2406.267112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2406.345873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2406.419354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2406.484580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2406.544206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2406.618551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2406.669102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2406.730312] LNet: 125325:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2406.771393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2406.813951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2406.853562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2406.933591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2406.974258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2407.019457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2414.671886] LNet: 125411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2414.685820] LNetError: 125411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2414.693778] LNet: Removed LNI 192.168.203.24@tcp [ 2414.718841] LNet: Removed LNI 192.168.203.24@tcp1 [ 2415.229079] Key type .llcrypt unregistered [ 2415.233907] Key type ._llcrypt unregistered [ 2435.178462] Key type ._llcrypt registered [ 2435.183296] Key type .llcrypt registered [ 2436.618754] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 01:49:48 (1787550588) [ 2437.236594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2437.263675] alg: No test for adler32 (adler32-zlib) [ 2438.196799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2438.263598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2438.291278] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2439.285613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2439.316356] LNet: Added LNI 192.168.203.24@tcp2 [8/256/0/180] [ 2440.534805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp [ 2440.705261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2441.761500] LNet: Removed LNI 192.168.203.24@tcp [ 2449.115551] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 01:50:01 (1787550601) [ 2450.647573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2452.705964] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 01:50:05 (1787550605) [ 2460.402627] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 01:50:12 (1787550612) [ 2467.607908] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 01:50:20 (1787550620) [ 2475.549115] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 01:50:27 (1787550627) [ 2484.073637] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 01:50:36 (1787550636) [ 2496.040305] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 01:50:47 (1787550647) [ 2506.727494] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 01:50:58 (1787550658) [ 2514.475523] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 01:51:07 (1787550667) [ 2522.266417] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 01:51:14 (1787550674) [ 2530.421170] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 01:51:22 (1787550682) [ 2537.606794] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 01:51:30 (1787550690) [ 2544.675147] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 01:51:37 (1787550697) [ 2551.592395] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 01:51:44 (1787550704) [ 2558.851662] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 01:51:51 (1787550711) [ 2566.879944] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 01:51:59 (1787550719) [ 2574.809267] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 01:52:07 (1787550727) [ 2581.911992] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 01:52:14 (1787550734) [ 2588.352081] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 01:52:20 (1787550740) [ 2588.449238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2588.492354] LNet: 137105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2588.504677] LNetError: 137105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2588.532922] LNet: Removed LNI 192.168.203.24@tcp2 [ 2588.570987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2588.711195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2588.733462] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2589.726665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2589.749915] LNet: 137254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2589.758858] LNetError: 137254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2589.777735] LNet: Removed LNI 192.168.203.24@tcp [ 2589.804209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2589.879326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2589.899479] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2596.049453] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 01:52:28 (1787550748) [ 2602.798258] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 01:52:35 (1787550755) [ 2608.869601] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 01:52:41 (1787550761) [ 2615.631068] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 01:52:48 (1787550768) [ 2623.951564] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 01:52:56 (1787550776) [ 2633.689846] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 01:53:06 (1787550786) [ 2642.658922] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 01:53:14 (1787550794) [ 2654.388938] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 01:53:26 (1787550806) [ 2667.370361] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 01:53:38 (1787550818) [ 2680.814405] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 01:53:52 (1787550832) [ 2690.236366] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 01:54:02 (1787550842) [ 2696.753286] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 01:54:09 (1787550849) [ 2704.399189] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2431 sec ========== 01:54:16 (1787550856) [ 2704.955579] LNet: 144724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2704.973085] LNetError: 144724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2705.002159] LNet: Removed LNI 192.168.203.24@tcp [ 2706.030483] Key type .llcrypt unregistered [ 2706.033530] Key type ._llcrypt unregistered [ 2722.162213] Key type ._llcrypt registered [ 2722.170399] Key type .llcrypt registered [ 2722.494287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2722.522088] alg: No test for adler32 (adler32-zlib) [ 2723.743587] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2724.075314] LNet: Added LNI 192.168.203.24@tcp [8/256/0/180] [ 2725.888219] Key type lgssc registered [ 2727.571066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2797.163792] Lustre: Mounted lustre-client - version 2.17.57_83_g6b3cfd8 [ 2801.817613] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2809.343506] Lustre: Unmounted lustre-client [ 2845.119855] Key type lgssc unregistered [ 2845.367298] LNet: 147682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2845.377706] LNetError: 147682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2845.392086] LNet: Removed LNI 192.168.203.24@tcp [ 2845.945165] Key type .llcrypt unregistered [ 2845.950065] Key type ._llcrypt unregistered