[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 452566550 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003224] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.007827] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010139] LSM: Security Framework initializing [ 0.011063] Yama: becoming mindful. [ 0.012054] SELinux: Initializing. [ 0.014077] *** VALIDATE selinux *** [ 0.022713] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026794] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029005] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] *** VALIDATE tmpfs *** [ 0.031000] *** VALIDATE proc *** [ 0.031249] *** VALIDATE cgroup *** [ 0.032012] *** VALIDATE cgroup2 *** [ 0.033282] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036031] Spectre V2 : User space: Vulnerable [ 0.037009] Speculative Store Bypass: Vulnerable [ 0.040018] debug: unmapping init [mem 0xffffffffb7059000-0xffffffffb7060fff] [ 0.042241] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043813] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044029] ... version: 2 [ 0.045015] ... bit width: 48 [ 0.046015] ... generic registers: 4 [ 0.047016] ... value mask: 0000ffffffffffff [ 0.048016] ... max period: 00007fffffffffff [ 0.049015] ... fixed-purpose events: 3 [ 0.050015] ... event mask: 000000070000000f [ 0.051331] rcu: Hierarchical SRCU implementation. [ 0.053612] smp: Bringing up secondary CPUs ... [ 0.054504] x86: Booting SMP configuration: [ 0.055028] .... node #0, CPUs: #1 #2 #3 [ 0.058170] smp: Brought up 1 node, 4 CPUs [ 0.060015] smpboot: Max logical packages: 1 [ 0.061025] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.136228] node 0 deferred pages initialised in 73ms [ 0.140114] devtmpfs: initialized [ 0.141327] x86/mm: Memory block size: 128MB [ 0.144887] gcov: version magic: 0x41383552 [ 0.148306] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151112] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154353] pinctrl core: initialized pinctrl subsystem [ 0.156235] [ 0.156758] ************************************************************* [ 0.160015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163012] ** ** [ 0.165013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168013] ** ** [ 0.170012] ** This means that this kernel is built to expose internal ** [ 0.172014] ** IOMMU data structures, which may compromise security on ** [ 0.174013] ** your system. ** [ 0.177014] ** ** [ 0.179013] ** If you see this message and you are not debugging the ** [ 0.182015] ** kernel, report this immediately to your vendor! ** [ 0.185023] ** ** [ 0.188013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.190012] ************************************************************* [ 0.193674] NET: Registered protocol family 16 [ 0.196570] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.201071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.205083] cpuidle: using governor menu [ 0.207482] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.208450] PCI: Using configuration type 1 for base access [ 0.209224] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218058] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219081] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.222111] cryptd: max_cpu_qlen set to 1000 [ 0.226342] ACPI: Added _OSI(Module Device) [ 0.228022] ACPI: Added _OSI(Processor Device) [ 0.229013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236367] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242515] ACPI: Interpreter enabled [ 0.243051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.244000] ACPI: Using IOAPIC for interrupt routing [ 0.247145] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.250487] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.261184] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.263050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.266023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.270183] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273687] acpiphp: Slot [2] registered [ 0.275277] acpiphp: Slot [5] registered [ 0.277127] acpiphp: Slot [6] registered [ 0.279122] acpiphp: Slot [3] registered [ 0.281119] acpiphp: Slot [4] registered [ 0.282110] acpiphp: Slot [7] registered [ 0.284110] acpiphp: Slot [8] registered [ 0.286144] acpiphp: Slot [9] registered [ 0.288096] acpiphp: Slot [10] registered [ 0.291134] acpiphp: Slot [11] registered [ 0.293101] acpiphp: Slot [12] registered [ 0.294101] acpiphp: Slot [13] registered [ 0.296154] acpiphp: Slot [14] registered [ 0.297117] acpiphp: Slot [15] registered [ 0.299120] acpiphp: Slot [16] registered [ 0.300097] acpiphp: Slot [17] registered [ 0.302113] acpiphp: Slot [18] registered [ 0.304108] acpiphp: Slot [19] registered [ 0.305132] acpiphp: Slot [20] registered [ 0.307116] acpiphp: Slot [21] registered [ 0.308190] acpiphp: Slot [22] registered [ 0.310102] acpiphp: Slot [23] registered [ 0.311205] acpiphp: Slot [24] registered [ 0.313110] acpiphp: Slot [25] registered [ 0.314126] acpiphp: Slot [26] registered [ 0.316097] acpiphp: Slot [27] registered [ 0.318086] acpiphp: Slot [28] registered [ 0.319113] acpiphp: Slot [29] registered [ 0.320113] acpiphp: Slot [30] registered [ 0.322113] acpiphp: Slot [31] registered [ 0.323062] PCI host bridge to bus 0000:00 [ 0.325025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.328030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.330023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.333023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.336024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.339029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.341164] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.345155] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.348477] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.357015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.361062] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.365021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.368018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.374560] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.377861] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.380052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.382679] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.388016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.399911] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.404016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.409000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.416021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.422018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.437018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.448827] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.454018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.459016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.472020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.482170] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.484406] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.487437] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.490409] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.492290] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.498194] iommu: Default domain type: Passthrough [ 0.501479] SCSI subsystem initialized [ 0.503186] ACPI: bus type USB registered [ 0.505147] usbcore: registered new interface driver usbfs [ 0.508129] usbcore: registered new interface driver hub [ 0.512123] usbcore: registered new device driver usb [ 0.514176] pps_core: LinuxPPS API ver. 1 registered [ 0.517016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.522068] PTP clock support registered [ 0.525145] EDAC MC: Ver: 3.0.0 [ 0.527401] PCI: Using ACPI for IRQ routing [ 0.530877] NetLabel: Initializing [ 0.533015] NetLabel: domain hash size = 128 [ 0.535011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.538090] NetLabel: unlabeled traffic allowed by default [ 0.541111] vgaarb: loaded [ 0.543295] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.545018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.553429] clocksource: Switched to clocksource kvm-clock [ 0.663618] VFS: Disk quotas dquot_6.6.0 [ 0.665793] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.669146] *** VALIDATE ramfs *** [ 0.670701] *** VALIDATE hugetlbfs *** [ 0.672656] pnp: PnP ACPI init [ 0.675319] pnp: PnP ACPI: found 6 devices [ 0.692159] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.695759] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.698240] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.700390] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.702562] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.705195] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.708124] NET: Registered protocol family 2 [ 0.711063] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.716631] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.721800] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.727576] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.731468] TCP: Hash tables configured (established 65536 bind 65536) [ 0.734712] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.738123] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.740668] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.743566] NET: Registered protocol family 1 [ 0.746551] RPC: Registered named UNIX socket transport module. [ 0.748625] RPC: Registered udp transport module. [ 0.750563] RPC: Registered tcp transport module. [ 0.752309] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.755058] NET: Registered protocol family 44 [ 0.756869] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.759319] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.761544] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.763945] PCI: CLS 0 bytes, default 64 [ 0.765689] Unpacking initramfs... [ 2.180542] debug: unmapping init [mem 0xffff925e7cc64000-0xffff925e7ffcffff] [ 2.184914] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.187404] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.190501] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.692434] Initialise system trusted keyrings [ 2.694023] Key type blacklist registered [ 2.695595] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.705627] zbud: loaded [ 2.708847] *** VALIDATE nfs *** [ 2.710212] *** VALIDATE nfs4 *** [ 2.711944] pstore: using deflate compression [ 2.719211] Platform Keyring initialized [ 2.833054] NET: Registered protocol family 38 [ 2.835369] Key type asymmetric registered [ 2.836990] Asymmetric key parser 'x509' registered [ 2.839559] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.842580] io scheduler mq-deadline registered [ 2.844820] io scheduler kyber registered [ 2.846615] io scheduler bfq registered [ 2.849052] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.851482] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.854029] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.856711] ACPI: Power Button [PWRF] [ 2.861616] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.869552] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.886171] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.913920] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.942368] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.947561] Non-volatile memory driver v1.3 [ 2.949328] Linux agpgart interface v0.103 [ 2.984154] virtio_blk virtio1: [vda] 146536 512-byte logical blocks (75.0 MB/71.6 MiB) [ 2.989381] vda: detected capacity change from 0 to 75026432 [ 3.005763] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.009023] vdb: detected capacity change from 0 to 1073741824 [ 3.017390] libphy: Fixed MDIO Bus: probed [ 3.022885] usbcore: registered new interface driver usbserial_generic [ 3.025544] usbserial: USB Serial support registered for generic [ 3.027614] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.031333] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.033077] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.035729] mousedev: PS/2 mouse device common for all mice [ 3.038532] rtc_cmos 00:05: RTC can wake from S4 [ 3.042063] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.042419] rtc_cmos 00:05: registered as rtc0 [ 3.047850] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.048617] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.051377] intel_pstate: CPU model not supported [ 3.056753] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.061232] hid: raw HID events driver (C) Jiri Kosina [ 3.063093] usbcore: registered new interface driver usbhid [ 3.064578] usbhid: USB HID core driver [ 3.065990] drop_monitor: Initializing network drop monitor service [ 3.067568] Initializing XFRM netlink socket [ 3.069296] NET: Registered protocol family 10 [ 3.072098] Segment Routing with IPv6 [ 3.073687] NET: Registered protocol family 17 [ 3.075726] mpls_gso: MPLS GSO support [ 3.082528] RAS: Correctable Errors collector initialized. [ 3.084838] AVX version of gcm_enc/dec engaged. [ 3.086577] AES CTR mode by8 optimization enabled [ 3.160634] sched_clock: Marking stable (3160612432, 0)->(4078888672, -918276240) [ 3.163798] registered taskstats version 1 [ 3.166348] Loading compiled-in X.509 certificates [ 3.168438] zswap: loaded using pool lzo/zbud [ 3.196292] Key type big_key registered [ 3.209095] Key type encrypted registered [ 3.210861] ima: No TPM chip found, activating TPM-bypass! [ 3.213606] ima: Allocated hash algorithm: sha1 [ 3.215643] ima: No architecture policies found [ 3.217814] evm: Initialising EVM extended attributes: [ 3.220088] evm: security.selinux [ 3.221401] evm: security.ima [ 3.222684] evm: security.capability [ 3.224069] evm: HMAC attrs: 0x1 [ 3.226613] rtc_cmos 00:05: setting system clock to 2026-08-22 02:16:48 UTC (1787365008) [ 3.233757] debug: unmapping init [mem 0xffffffffb8003000-0xffffffffb81fffff] [ 3.237164] debug: unmapping init [mem 0xffffffffb6d82000-0xffffffffb7058fff] [ 3.246148] Write protecting the kernel read-only data: 28672k [ 3.249871] debug: unmapping init [mem 0xffffffffb5403000-0xffffffffb55fffff] [ 3.252779] debug: unmapping init [mem 0xffffffffb5d14000-0xffffffffb5dfffff] [ 3.292149] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.301117] systemd[1]: Detected virtualization kvm. [ 3.302965] systemd[1]: Detected architecture x86-64. [ 3.305246] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.334516] systemd[1]: No hostname configured. [ 3.336383] systemd[1]: Set hostname to . [ 3.338625] random: systemd: uninitialized urandom read (16 bytes read) [ 3.341242] systemd[1]: Initializing machine ID from random generator. [ 3.393997] random: ln: uninitialized urandom read (6 bytes read) [ 3.492923] random: systemd: uninitialized urandom read (16 bytes read) [ 3.496783] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.505177] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.517324] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.250852] device-mapper: uevent: version 1.0.3 [ 4.253057] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.044273] virtio_net virtio0 ens2: renamed from eth0 [ 5.136960] scsi host0: ata_piix [ 5.146774] scsi host1: ata_piix [ 5.148481] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.150516] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.551625] dracut-initqueue[594]: RTNETLINK answers: File exists [ 9.730823] random: crng init done [ 9.732414] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.283287] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.510980] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.791704] SELinux: Disabled at runtime. [ 11.854493] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.862211] systemd[1]: Detected virtualization kvm. [ 11.864136] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.426115] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.429151] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.433956] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.438669] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.441129] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.447340] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.452485] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 12.526036] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.966907] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.272211] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.315379] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.427711] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.438242] EDAC sbridge: Ver: 1.1.2 [ 14.486553] Key type dns_resolver registered [ 14.816239] NFS: Registering the id_resolver key type [ 14.817877] Key type id_resolver registered [ 14.819770] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg431-client login: [ 70.895203] libcfs: loading out-of-tree module taints kernel. [ 71.021367] Key type ._llcrypt registered [ 71.031580] Key type .llcrypt registered [ 71.707476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 71.733352] alg: No test for adler32 (adler32-zlib) [ 73.541696] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 74.679281] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 76.536298] Key type lgssc registered [ 78.600749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 110.887115] hrtimer: interrupt took 3106295 ns [ 221.612119] Lustre: Mounted lustre-client - version 2.17.57_46_g5381a3b [ 227.492270] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.444341] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing check_logdir /tmp/testlogs/ [ 247.267696] Lustre: lustre-OST0000-osc-ffff925ed16e4000: disconnect after 23s idle [ 248.751641] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing yml_node [ 253.510520] Lustre: DEBUG MARKER: Client: 2.17.57.46 [ 256.390972] Lustre: DEBUG MARKER: MDS: 2.17.57.46 [ 259.067404] Lustre: DEBUG MARKER: OSS: 2.17.57.46 [ 261.121137] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Aug 21 22:21:04 EDT 2026 [ 278.967628] Lustre: Unmounted lustre-client [ 298.751140] Key type lgssc unregistered [ 299.029873] LNet: 8415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.035739] LNetError: 8415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 299.073891] LNet: Removed LNI 192.168.204.31@tcp [ 300.105753] Key type .llcrypt unregistered [ 300.112779] Key type ._llcrypt unregistered [ 328.957850] Key type ._llcrypt registered [ 328.960282] Key type .llcrypt registered [ 329.252294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 329.273040] alg: No test for adler32 (adler32-zlib) [ 330.242087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 330.323631] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 330.344446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 334.509402] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 334.640224] LNet: 9298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.653530] LNetError: 9298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 334.668581] LNet: Removed LNI 192.168.204.31@tcp [ 335.312935] Key type .llcrypt unregistered [ 335.314763] Key type ._llcrypt unregistered [ 342.084763] Key type ._llcrypt registered [ 342.086428] Key type .llcrypt registered [ 343.613739] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 22:22:27 (1787365347) [ 344.039328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 344.054256] alg: No test for adler32 (adler32-zlib) [ 344.998438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.068666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.083843] LNet: 9722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.101157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.153679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 345.177601] LNet: 9729:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 351.951766] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 22:22:35 (1787365355) [ 352.047619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.078689] LNet: 10301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.125845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.240993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 359.561959] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 22:22:43 (1787365363) [ 359.856871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.886408] LNet: 10894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.906491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.978258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 367.188741] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 22:22:50 (1787365370) [ 367.441578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.471395] LNet: 11481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.540394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.703977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 374.895869] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 22:22:58 (1787365378) [ 375.004530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.051411] LNet: 12060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.071238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.159224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 375.514851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 382.797253] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 22:23:06 (1787365386) [ 382.908680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.997650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.119172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 383.393971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 383.861434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 393.918651] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 22:23:15 (1787365395) [ 394.162465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.246249] LNet: 13247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.251898] LNet: 13247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 394.329252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.516800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 403.685832] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 22:23:27 (1787365407) [ 403.975514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.023840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.104823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 404.153660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 404.921918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 404.985808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 405.585426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 405.647962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 406.043865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 406.105683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 406.453932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 406.509546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 407.096509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 407.136822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 407.889931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 407.940576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 408.773170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 408.852809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 409.273835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 409.329378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 409.996373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 410.079055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 410.652600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 410.726468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 411.496714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 411.604586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 412.472502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 412.502423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 423.034903] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 22:23:46 (1787365426) [ 423.147261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.178398] LNet: 14511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 423.185131] LNet: 14511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 423.223170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.335542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 423.417899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 430.632657] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 22:23:54 (1787365434) [ 430.748115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.861566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.026979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 431.105196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 438.156887] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 22:24:01 (1787365441) [ 438.504872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.546085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.651820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 438.694354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 444.872911] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 22:24:09 (1787365449) [ 445.196860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.227069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.335326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 445.383815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 453.037304] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 22:24:16 (1787365456) [ 453.170263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.227115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.326846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 453.380452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 460.385856] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 22:24:24 (1787365464) [ 460.512568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.546395] LNet: 17445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.555226] LNet: 17445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 460.576241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.677583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 460.717654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 468.693281] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 22:24:32 (1787365472) [ 468.815818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.883573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.069443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.718962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 476.870994] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 22:24:40 (1787365480) [ 476.984237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.051859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.342609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 477.613758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 485.019235] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 22:24:48 (1787365488) [ 485.142233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.221410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.423858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 485.593649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 494.174453] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 22:24:57 (1787365497) [ 494.513967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.541930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.156294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.369572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 503.472092] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 22:25:06 (1787365506) [ 503.595356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.667527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.798980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 503.851504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 512.466479] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 22:25:15 (1787365515) [ 512.577044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.641039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.773570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 512.846581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 520.116715] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 22:25:24 (1787365524) [ 520.472126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.521687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.669681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 527.720374] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 22:25:31 (1787365531) [ 528.020400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.045854] LNet: 22305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.051164] LNet: 22305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 528.083786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.199198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 535.063988] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 22:25:38 (1787365538) [ 535.320451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.367720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.494649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 536.051895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 543.358611] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 22:25:47 (1787365547) [ 543.734984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.782551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.900972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 543.935885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 551.244942] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 22:25:55 (1787365555) [ 551.457797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.511366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.584123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 551.635464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 559.799979] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 22:26:03 (1787365563) [ 559.969147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.014403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.100970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 560.147368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 568.057282] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 22:26:11 (1787365571) [ 568.546885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.605572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.713613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 568.769859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 575.991265] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 22:26:19 (1787365579) [ 576.155339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.193816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.267747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 576.354326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 576.375922] LNetError: 25837:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 576.437113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 576.467300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 582.522175] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 22:26:26 (1787365586) [ 582.614355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.662907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.753543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.iegQVgp2t2 [ 582.814374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.iegQVgp2t2 [ 582.862293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.iegQVgp2t2 [ 589.716931] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 22:26:33 (1787365593) [ 589.839732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.898046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.979972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 590.024621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 590.076408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 596.301749] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 22:26:40 (1787365600) [ 596.712075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.793987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.879317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 596.912608] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 598.671080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 600.320909] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 602.304352] Key type lgssc registered [ 603.644718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 607.825759] LNetError: 27598:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.31@tcp99 has no available nets [ 610.388277] Key type lgssc unregistered [ 610.736725] LNetError: 28709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 610.751639] LNet: Removed LNI 192.168.204.31@tcp [ 611.559688] Key type .llcrypt unregistered [ 611.565151] Key type ._llcrypt unregistered [ 619.269551] Key type ._llcrypt registered [ 619.274257] Key type .llcrypt registered [ 620.745253] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 22:27:04 (1787365624) [ 621.350839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 621.370626] alg: No test for adler32 (adler32-zlib) [ 622.370659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.469811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 625.605193] LNet: Message forwarding will be enabled in 2s [ 628.704944] LNet: Message forwarding enabled [ 629.237386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 629.278261] LNet: Message forwarding will stop in 2s to 6s [ 634.086853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 634.154286] LNet: Message forwarding disabled [ 636.693741] LNet: Message forwarding will be enabled in 2s [ 640.288176] LNet: Message forwarding enabled [ 643.916747] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 22:27:27 (1787365647) [ 644.297903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.341401] LNet: 30119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.888799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.982293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 653.392610] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 22:27:37 (1787365657) [ 653.626937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.652319] LNet: 30701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.685192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.734190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 653.791820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 653.886352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 653.925866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 653.956196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 653.986222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 654.014108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 654.046790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 654.079502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 654.110522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 654.162434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 654.208132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 654.249583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 654.302641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 654.339917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 654.389867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 654.429134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 654.479716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 654.527427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 654.572116] LNetError: 30744:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 662.945176] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 22:27:46 (1787365666) [ 663.053762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.146337] LNet: 31308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.203273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.320313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 673.768306] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 22:27:56 (1787365676) [ 676.838922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg431-server [ 679.204695] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 22:28:02 (1787365682) [ 679.436760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.545456] LNet: 32244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.645939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.910159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 679.976616] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 681.070499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[32-36]@tcp [ 681.115576] LNet: 32383:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 681.151518] LNetError: 32260:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.34@tcp has gone from up to down [ 681.200511] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.32@tcp is being used as a gateway but routing feature is not turned on [ 682.178889] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.33@tcp is being used as a gateway but routing feature is not turned on [ 682.211995] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 683.192514] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.32@tcp is being used as a gateway but routing feature is not turned on [ 683.226354] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 685.227737] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.32@tcp is being used as a gateway but routing feature is not turned on [ 685.242470] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 689.326141] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.35@tcp is being used as a gateway but routing feature is not turned on [ 689.344786] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 17 previous similar messages [ 690.523890] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 22:28:14 (1787365694) [ 693.036704] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg431-server [ 695.322031] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 22:28:18 (1787365698) [ 695.621658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.668644] LNet: 33319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.682814] LNetError: 33319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 695.716320] LNet: Removed LNI 192.168.204.31@tcp [ 695.774902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.939563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 695.983149] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 696.498634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[32-37/2]@tcp [ 696.547679] LNet: 33436:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 696.557651] LNet: 33436:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 696.590906] LNetError: 33323:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.32@tcp has gone from up to down [ 696.614728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[32-37/2]@tcp [ 696.621256] LNetError: 33323:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 702.832916] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 22:28:26 (1787365706) [ 703.287085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.337288] LNetError: 34030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.360430] LNet: Removed LNI 192.168.204.31@tcp [ 703.399105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.535358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 703.634720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.686626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.758426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 703.874752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.896883] LNet: 34056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.903678] LNet: 34056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 703.923535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.003725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 704.102950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.130717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.185402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 704.255764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.286171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.333456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 704.427408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.472574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.533416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 704.598351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.638561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.701677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 704.793460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.829817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.910880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 704.982174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.021994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.114176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 705.241621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.329719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.423479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 705.552105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.609163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.661730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 705.722979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.756910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.833863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 712.088691] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 22:28:36 (1787365716) [ 713.824269] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg431-server [ 715.729728] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 22:28:39 (1787365719) [ 717.428666] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg431-server [ 719.264926] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 22:28:43 (1787365723) [ 720.027305] Key type .llcrypt unregistered [ 720.034921] Key type ._llcrypt unregistered [ 725.877122] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 726.060106] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 726.067896] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 726.081228] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 727.534594] Key type ._llcrypt registered [ 727.537344] Key type .llcrypt registered [ 727.695641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 727.719385] alg: No test for adler32 (adler32-zlib) [ 728.675214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.743960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 728.771840] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 729.607289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 729.627834] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 731.498427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 731.532240] LNet: Removed LNI 10.1.2.3@tcp [ 731.646815] LNet: 36148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 731.656666] LNetError: 36148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 731.676876] LNet: Removed LNI 192.168.204.31@tcp [ 732.389222] Key type .llcrypt unregistered [ 732.391853] Key type ._llcrypt unregistered [ 743.895358] Key type ._llcrypt registered [ 743.897335] Key type .llcrypt registered [ 744.955298] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 22:29:09 (1787365749) [ 745.570502] Key type .llcrypt unregistered [ 745.575655] Key type ._llcrypt unregistered [ 751.601632] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 751.685441] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 751.689785] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 751.693685] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 754.308297] Key type ._llcrypt registered [ 754.310203] Key type .llcrypt registered [ 754.508864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 754.529477] alg: No test for adler32 (adler32-zlib) [ 755.464121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.515903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 755.544123] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 756.316438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 756.343791] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 759.141571] LNet: 37650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.147801] LNetError: 37650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 759.153573] LNet: Removed LNI 10.1.2.3@tcp [ 759.170899] LNet: Removed LNI 192.168.204.31@tcp [ 759.926613] Key type .llcrypt unregistered [ 759.931800] Key type ._llcrypt unregistered [ 771.047678] Key type ._llcrypt registered [ 771.054191] Key type .llcrypt registered [ 772.455669] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 22:29:36 (1787365776) [ 773.015393] Key type .llcrypt unregistered [ 773.017650] Key type ._llcrypt unregistered [ 779.674476] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 779.765807] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 779.797177] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 779.811083] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 782.370474] Key type ._llcrypt registered [ 782.379566] Key type .llcrypt registered [ 782.586881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 782.610402] alg: No test for adler32 (adler32-zlib) [ 783.538404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.631579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 783.675194] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 785.476361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 785.494196] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 787.633819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 787.664273] LNet: Removed LNI 10.1.2.3@tcp [ 787.686799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 787.733250] LNet: Removed LNI 10.1.2.31@tcp [ 787.757866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 787.793678] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 788.524274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 788.544899] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 789.406303] LNet: 39577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.414483] LNetError: 39577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 789.419243] LNet: Removed LNI 10.1.2.31@tcp [ 789.436334] LNet: Removed LNI 10.1.2.3@tcp [ 790.201660] Key type .llcrypt unregistered [ 790.204189] Key type ._llcrypt unregistered [ 804.341228] Key type ._llcrypt registered [ 804.343911] Key type .llcrypt registered [ 805.824782] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 22:30:09 (1787365809) [ 806.297469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 806.316723] alg: No test for adler32 (adler32-zlib) [ 807.322402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.406922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 807.446516] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 808.386648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.432382] LNet: 40534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.452441] LNetError: 40534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.468987] LNet: Removed LNI 192.168.204.31@tcp [ 808.496404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.578864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.31@tcp [ 808.630078] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 809.412676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.31@tcp [ 809.460194] LNet: Removed LNI 192.168.204.31@tcp [ 809.546840] LNet: 40669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.409441] Key type .llcrypt unregistered [ 810.411076] Key type ._llcrypt unregistered [ 821.670719] Key type ._llcrypt registered [ 821.682537] Key type .llcrypt registered [ 822.912695] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 22:30:26 (1787365826) [ 824.923855] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg431-server [ 826.875766] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 22:30:30 (1787365830) [ 827.654329] Key type .llcrypt unregistered [ 827.655939] Key type ._llcrypt unregistered [ 834.274123] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 834.537790] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 834.549471] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 834.571024] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 837.229431] Key type ._llcrypt registered [ 837.234358] Key type .llcrypt registered [ 837.456704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 837.490235] alg: No test for adler32 (adler32-zlib) [ 838.597393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.744910] LNet: Added LNI 192.168.204.31@tcp1 [8/256/0/180] [ 839.501956] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 841.947803] LNet: 42513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.956843] LNetError: 42513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.962079] LNet: Removed LNI 192.168.204.31@tcp1 [ 841.978720] LNet: Removed LNI 10.1.2.3@tcp2 [ 842.489430] Key type .llcrypt unregistered [ 842.491536] Key type ._llcrypt unregistered [ 857.048343] Key type ._llcrypt registered [ 857.051647] Key type .llcrypt registered [ 858.288752] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 22:31:02 (1787365862) [ 858.692184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 858.702899] alg: No test for adler32 (adler32-zlib) [ 859.771342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.824563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 859.841544] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 860.676782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.699123] LNet: 43487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.710489] LNetError: 43487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.743369] LNet: Removed LNI 192.168.204.31@tcp [ 860.796822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.887892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 860.930806] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 867.357211] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 22:31:11 (1787365871) [ 867.529262] LNet: 44072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.538964] LNetError: 44072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.578539] LNet: Removed LNI 192.168.204.31@tcp [ 868.524690] Key type .llcrypt unregistered [ 868.532573] Key type ._llcrypt unregistered [ 875.067370] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 875.327166] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 875.341212] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 875.353468] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 878.090044] Key type ._llcrypt registered [ 878.096408] Key type .llcrypt registered [ 878.356978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 878.391663] alg: No test for adler32 (adler32-zlib) [ 879.408468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.498698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 879.524862] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 880.169574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 880.189662] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 882.369925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.393871] LNet: 44827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.403910] LNetError: 44827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 882.413359] LNet: Removed LNI 10.1.2.3@tcp [ 882.442266] LNet: Removed LNI 192.168.204.31@tcp [ 882.473986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.548573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 882.594465] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 882.600959] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 883.186548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.212462] LNet: 44854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.223398] LNetError: 44854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 883.233968] LNetError: 44854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 883.245283] LNet: Removed LNI 10.1.2.3@tcp [ 883.253527] LNet: Removed LNI 192.168.204.31@tcp [ 883.274897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.329846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 883.348677] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 883.356593] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 883.805422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 883.826491] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 884.930364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.949942] LNet: 45016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.954498] LNetError: 45016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 884.963267] LNetError: 45016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 884.969570] LNet: Removed LNI 10.1.2.3@tcp [ 884.972047] LNet: Removed LNI 192.168.204.31@tcp [ 884.986908] LNet: Removed LNI 192.168.204.31@tcp2 [ 885.007676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.067912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 885.096076] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 885.100136] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 885.105892] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 893.734398] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 22:31:36 (1787365896) [ 893.859776] LNet: 45605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.868869] LNetError: 45605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 893.879641] LNetError: 45605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 893.891390] LNet: Removed LNI 10.1.2.3@tcp [ 893.899164] LNet: Removed LNI 192.168.204.31@tcp [ 893.932307] LNet: Removed LNI 192.168.204.31@tcp2 [ 895.110831] Key type .llcrypt unregistered [ 895.114233] Key type ._llcrypt unregistered [ 901.705358] Key type ._llcrypt registered [ 901.707932] Key type .llcrypt registered [ 901.944066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 901.959354] alg: No test for adler32 (adler32-zlib) [ 902.948911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.034194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 903.052985] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 904.195979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.231468] LNet: 46014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.249061] LNetError: 46014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.308163] LNet: Removed LNI 192.168.204.31@tcp [ 904.384587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.488810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 904.536543] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 905.280482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.326055] LNet: 46044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.359575] LNetError: 46044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.389786] LNet: Removed LNI 192.168.204.31@tcp [ 905.433520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.494816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 905.520531] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 905.870939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.916538] LNet: Removed LNI 192.168.204.31@tcp [ 905.938969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.998530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 906.018000] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 906.694224] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 906.814815] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 906.822583] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 906.830772] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 909.297172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.335648] LNet: 46153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.349757] LNet: 46153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 909.361995] LNetError: 46153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.367161] LNetError: 46153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 909.388757] LNet: Removed LNI 192.168.204.31@tcp [ 909.411968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.497321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 909.533067] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 916.910470] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 22:32:00 (1787365920) [ 917.166809] LNet: 46750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.171525] LNetError: 46750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.185869] LNet: Removed LNI 192.168.204.31@tcp [ 917.803813] Key type .llcrypt unregistered [ 917.808738] Key type ._llcrypt unregistered [ 924.616302] Key type ._llcrypt registered [ 924.618663] Key type .llcrypt registered [ 924.822447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 924.831800] alg: No test for adler32 (adler32-zlib) [ 925.764696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.824301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 925.845252] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 926.699095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.730557] LNet: 47169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.738644] LNetError: 47169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.762425] LNet: Removed LNI 192.168.204.31@tcp [ 926.793184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.930643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 926.986809] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 934.207542] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 22:32:17 (1787365937) [ 934.308622] LNet: 47754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.323520] LNetError: 47754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.353455] LNet: Removed LNI 192.168.204.31@tcp [ 934.986451] Key type .llcrypt unregistered [ 934.989111] Key type ._llcrypt unregistered [ 942.224637] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 942.439931] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 942.447072] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 942.453107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 945.044577] Key type ._llcrypt registered [ 945.046667] Key type .llcrypt registered [ 945.234823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 945.260912] alg: No test for adler32 (adler32-zlib) [ 946.192449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.246950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 946.274752] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 947.104465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.127566] LNet: 48215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.133765] LNetError: 48215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.147392] LNet: Removed LNI 192.168.204.31@tcp [ 947.187652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.266436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 947.290383] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 953.972887] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 22:32:37 (1787365957) [ 954.172220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.220523] LNet: 48810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.233595] LNetError: 48810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.269758] LNet: Removed LNI 192.168.204.31@tcp [ 954.334772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.473548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 961.204406] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 22:32:45 (1787365965) [ 961.581432] LNet: 49396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.324865] Key type .llcrypt unregistered [ 962.328839] Key type ._llcrypt unregistered [ 969.143508] Key type ._llcrypt registered [ 969.145867] Key type .llcrypt registered [ 969.348775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 969.367988] alg: No test for adler32 (adler32-zlib) [ 970.317670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.379538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 970.448829] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 971.554418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.616178] LNet: 49797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.628293] LNetError: 49797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.665971] LNet: Removed LNI 192.168.204.31@tcp [ 971.726281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.852844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 971.882905] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 979.750690] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 22:33:03 (1787365983) [ 980.286859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.322758] LNet: 50391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.333245] LNetError: 50391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.356575] LNet: Removed LNI 192.168.204.31@tcp [ 980.392712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.505300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 986.596126] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 22:33:10 (1787365990) [ 988.066920] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 990.036596] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 22:33:14 (1787365994) [ 991.419340] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 993.290261] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 22:33:17 (1787365997) [ 994.849412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 996.852843] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 22:33:20 (1787366000) [ 998.490767] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1000.447387] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 22:33:24 (1787366004) [ 1001.854636] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1003.801540] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 22:33:27 (1787366007) [ 1003.901952] LNet: 52436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.690736] Key type .llcrypt unregistered [ 1004.695445] Key type ._llcrypt unregistered [ 1010.615990] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1010.789909] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1010.793709] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1010.806484] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1013.455845] Key type ._llcrypt registered [ 1013.461244] Key type .llcrypt registered [ 1013.680567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1013.695449] alg: No test for adler32 (adler32-zlib) [ 1014.637930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.763698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1014.816860] LNet: Added LNI 192.168.204.31@tcp [8/512/0/180] [ 1014.832518] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1014.840343] LNet: 52770:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1014.974955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1015.022955] LNet: 52789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.031527] LNetError: 52789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1015.054098] LNet: Removed LNI 10.1.2.3@tcp [ 1015.060520] LNetError: 52789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.084264] LNet: Removed LNI 192.168.204.31@tcp [ 1015.129735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.213581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1015.233652] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1015.312699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1015.330590] LNet: 52813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.334716] LNetError: 52813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.362318] LNet: Removed LNI 192.168.204.31@tcp [ 1015.397303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.462480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1015.487562] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1015.493058] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1015.496043] LNet: 52821:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1015.592439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1015.616660] LNet: 52839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.633531] LNetError: 52839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1015.640557] LNet: Removed LNI 10.1.2.3@tcp [ 1015.642844] LNetError: 52839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.665996] LNet: Removed LNI 192.168.204.31@tcp [ 1015.688541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.735529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1015.756413] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1015.766619] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1015.774288] LNet: 52847:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1016.035542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1016.065740] LNet: 52873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.073576] LNetError: 52873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1016.084700] LNet: Removed LNI 10.1.2.3@tcp [ 1016.088625] LNetError: 52873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.108742] LNet: Removed LNI 192.168.204.31@tcp [ 1016.130476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.243924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1016.286379] LNet: Added LNI 192.168.204.31@tcp [8/256/0/182] [ 1016.293730] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1016.472478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1016.504778] LNet: 52901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.514977] LNetError: 52901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1016.525990] LNet: Removed LNI 10.1.2.3@tcp [ 1016.529772] LNetError: 52901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.545932] LNet: Removed LNI 192.168.204.31@tcp [ 1016.564784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.640851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1016.666679] LNet: Added LNI 192.168.204.31@tcp [8/256/0/182] [ 1016.676444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1016.679375] LNet: 52909:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1016.757656] LNet: Removed LNI 10.1.2.3@tcp [ 1016.774502] LNet: Removed LNI 192.168.204.31@tcp [ 1017.531849] Key type .llcrypt unregistered [ 1017.537901] Key type ._llcrypt unregistered [ 1028.657274] Key type ._llcrypt registered [ 1028.661063] Key type .llcrypt registered [ 1030.028984] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 22:33:53 (1787366033) [ 1032.004225] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1034.259676] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 22:33:57 (1787366037) [ 1034.802873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1034.824534] alg: No test for adler32 (adler32-zlib) [ 1035.883726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1035.997319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1036.073931] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1037.243415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.31 [ 1037.258250] LNet: 54178:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1037.267711] LNet: Removed LNI 192.168.204.31@tcp [ 1044.017177] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 22:34:07 (1787366047) [ 1044.287050] LNet: 54747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.293256] LNetError: 54747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.336794] LNet: Removed LNI 192.168.204.31@tcp [ 1045.122365] Key type .llcrypt unregistered [ 1045.129092] Key type ._llcrypt unregistered [ 1051.809630] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1052.069561] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1054.853390] Key type ._llcrypt registered [ 1054.855990] Key type .llcrypt registered [ 1055.005223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1055.015817] alg: No test for adler32 (adler32-zlib) [ 1056.314205] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1062.452716] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 22:34:26 (1787366066) [ 1062.613593] LNet: 55658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.627909] LNetError: 55658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1062.646310] LNet: Removed LNI 10.1.2.3@tcp [ 1063.481692] Key type .llcrypt unregistered [ 1063.484419] Key type ._llcrypt unregistered [ 1070.260338] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1070.455609] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1070.459535] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1070.464538] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1072.159863] Key type ._llcrypt registered [ 1072.166803] Key type .llcrypt registered [ 1072.337846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1072.350423] alg: No test for adler32 (adler32-zlib) [ 1073.296155] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1080.071872] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 22:34:43 (1787366083) [ 1081.785634] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1083.503941] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 22:34:47 (1787366087) [ 1083.865442] LNet: 56845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.876116] LNetError: 56845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1083.909175] LNet: Removed LNI 10.1.2.3@tcp [ 1084.558236] Key type .llcrypt unregistered [ 1084.560284] Key type ._llcrypt unregistered [ 1090.811128] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1090.986853] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1090.990574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1090.993954] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1093.777828] Key type ._llcrypt registered [ 1093.780564] Key type .llcrypt registered [ 1093.993367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1094.012463] alg: No test for adler32 (adler32-zlib) [ 1094.999684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1095.075739] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1103.321967] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 22:35:07 (1787366107) [ 1103.809248] LNet: 58078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.815913] LNetError: 58078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1103.830227] LNet: Removed LNI 10.1.2.3@tcp [ 1104.584828] Key type .llcrypt unregistered [ 1104.588131] Key type ._llcrypt unregistered [ 1111.536103] Key type ._llcrypt registered [ 1111.542070] Key type .llcrypt registered [ 1111.823239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1111.851080] alg: No test for adler32 (adler32-zlib) [ 1112.733312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1112.752067] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1121.636463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.131@tcp [ 1122.429854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_interrupt -r 1 [ 1122.476324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_interrupt -r 1 [ 1122.531817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1122.557903] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1122.799227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1122.859899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1122.926617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.977692] LNet: 1 local NIs in recovery (showing 1): 192.168.204.31@tcp [ 1122.992841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1123.015449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1123.069951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1123.110842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1123.170259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1123.195126] LNet: 58520:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1123.211759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1123.243629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1123.268607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1123.301687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1123.355403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.387238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1123.421872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1123.918418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_dropped -r 1 [ 1123.958286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_dropped -r 1 [ 1123.994636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1124.014345] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1124.247801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.280800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1124.313066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.339970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.365854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.401921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.460979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.496909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.521871] LNet: 58630:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.550854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.575657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.612276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.670572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.717514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.779469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.827302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1125.258764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_aborted -r 1 [ 1125.306745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_aborted -r 1 [ 1125.344322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1125.373631] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1125.607596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1125.636419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1125.670397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1125.692162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1125.712816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1125.736514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1125.759383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1125.789752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1125.805430] LNet: 58739:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1125.819407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1125.849667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1125.872548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1125.908769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1125.948930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1125.987602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.019533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1126.485603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_no_route -r 1 [ 1126.519771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_no_route -r 1 [ 1126.544167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1126.762736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.792184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1126.819554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.844404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.870911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.895609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.935259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.989344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1127.034857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1127.060576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1127.086761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1127.122334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.147252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1127.190192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1127.224336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1127.693386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_timeout -r 1 [ 1127.737810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_timeout -r 1 [ 1127.778113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1127.807190] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1127.814060] LNet: Skipped 2 previous similar messages [ 1128.014767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1128.042736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1128.072177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1128.095459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1128.124272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1128.147741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1128.176919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1128.198942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1128.210900] LNet: 58957:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1128.219899] LNet: 58957:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1128.244797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1128.288427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1128.335352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1128.376099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1128.406354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1128.434530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1128.484743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1128.947801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_error -r 1 [ 1128.988896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_error -r 1 [ 1129.028663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1129.407565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1129.459795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1129.515451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1129.563769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1129.614860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1129.663420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1129.693482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1129.738682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1129.771310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1129.800884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1129.837467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1129.874449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1129.917860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.960314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.988251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1130.335314] LNet: 59111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1130.345655] LNetError: 59111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1130.356134] LNet: Removed LNI 192.168.204.31@tcp [ 1130.754451] Key type .llcrypt unregistered [ 1130.758780] Key type ._llcrypt unregistered [ 1149.038325] Key type ._llcrypt registered [ 1149.040120] Key type .llcrypt registered [ 1150.389682] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 22:35:54 (1787366154) [ 1151.294429] Key type .llcrypt unregistered [ 1151.300339] Key type ._llcrypt unregistered [ 1158.069920] Key type ._llcrypt registered [ 1158.077666] Key type .llcrypt registered [ 1158.356329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1158.374488] alg: No test for adler32 (adler32-zlib) [ 1159.350870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1159.397752] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1167.608509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.131@tcp [ 1178.066729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1178.115083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1178.144812] LNet: Added LNI 192.168.204.31@tcp1 [8/256/0/180] [ 1179.575879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_interrupt -r 1 [ 1179.631192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e local_interrupt -r 1 [ 1179.665319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_interrupt -r 1 [ 1179.694796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e local_interrupt -r 1 [ 1179.727888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e local_interrupt -r 1 [ 1179.759620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e local_interrupt -r 1 [ 1179.791640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e local_interrupt -r 1 [ 1179.823623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e local_interrupt -r 1 [ 1179.856123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1179.876917] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1179.886382] LNet: 1 local NIs in recovery (showing 1): 192.168.204.31@tcp [ 1180.105316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.126842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1180.154784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1180.177675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1180.206631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1180.234847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.290106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.328237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.343722] LNet: 60582:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1180.358822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.390526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.429699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.469765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.518191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.588430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.655452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1181.282650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_dropped -r 1 [ 1181.353553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e local_dropped -r 1 [ 1181.419483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_dropped -r 1 [ 1181.465160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e local_dropped -r 1 [ 1181.529419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e local_dropped -r 1 [ 1181.602344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e local_dropped -r 1 [ 1181.648737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e local_dropped -r 1 [ 1181.695600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e local_dropped -r 1 [ 1181.736553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1181.751861] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1181.766197] LNet: Skipped 2 previous similar messages [ 1182.017717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.042543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.077584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.105822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.139708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.169424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.215307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.247899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.276637] LNet: 60704:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.301185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.331135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.357867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.388872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.425586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.491333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.537727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1183.172983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_aborted -r 1 [ 1183.216947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e local_aborted -r 1 [ 1183.253443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_aborted -r 1 [ 1183.298237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e local_aborted -r 1 [ 1183.351070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e local_aborted -r 1 [ 1183.407205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e local_aborted -r 1 [ 1183.467790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e local_aborted -r 1 [ 1183.524633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e local_aborted -r 1 [ 1183.593662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1183.648006] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1183.650796] LNet: Skipped 2 previous similar messages [ 1183.951254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.978398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.022663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.050794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.097061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.150150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.179463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.202616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.221452] LNet: 60826:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1184.236304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.267868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.291666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1184.319170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.347145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.385486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.452466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1185.002556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_no_route -r 1 [ 1185.040582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e local_no_route -r 1 [ 1185.090757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_no_route -r 1 [ 1185.133113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e local_no_route -r 1 [ 1185.169412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e local_no_route -r 1 [ 1185.208279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e local_no_route -r 1 [ 1185.234454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e local_no_route -r 1 [ 1185.258804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e local_no_route -r 1 [ 1185.292905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1185.506612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1185.533754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1185.568582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1185.604262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1185.646946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1185.688713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1185.730906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1185.791778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1185.841922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1185.894975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1185.918975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1185.942930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1185.964592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.000855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.040556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1186.586383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_timeout -r 1 [ 1186.642941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e local_timeout -r 1 [ 1186.695482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_timeout -r 1 [ 1186.734275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e local_timeout -r 1 [ 1186.760918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e local_timeout -r 1 [ 1186.785393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e local_timeout -r 1 [ 1186.819260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e local_timeout -r 1 [ 1186.872887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e local_timeout -r 1 [ 1186.936611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1186.953608] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1186.962209] LNet: Skipped 5 previous similar messages [ 1187.176583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1187.208736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1187.249894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1187.296300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1187.348664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1187.393198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1187.448221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1187.499549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1187.520944] LNet: 61068:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1187.533103] LNet: 61068:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1187.557448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1187.592711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1187.660581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.697490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1187.741869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1187.784313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1187.827223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1188.338734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_error -r 1 [ 1188.395903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e local_error -r 1 [ 1188.462420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_error -r 1 [ 1188.515474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e local_error -r 1 [ 1188.548941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e local_error -r 1 [ 1188.601553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e local_error -r 1 [ 1188.660944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e local_error -r 1 [ 1188.718281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e local_error -r 1 [ 1188.779415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1189.029680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.049510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1189.077627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1189.111496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1189.133149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1189.161615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1189.190573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1189.213863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1189.244554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1189.296769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1189.371982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1189.422549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1189.461322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.491147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.515924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1194.697944] LNet: 61269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1194.704808] LNetError: 61269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1195.746440] LNet: Removed LNI 192.168.204.31@tcp [ 1195.748471] LNetError: 61269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1195.765880] LNet: Removed LNI 192.168.204.31@tcp1 [ 1196.287560] Key type .llcrypt unregistered [ 1196.294448] Key type ._llcrypt unregistered [ 1212.907272] Key type ._llcrypt registered [ 1212.910866] Key type .llcrypt registered [ 1214.239976] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 22:36:58 (1787366218) [ 1214.988331] Key type .llcrypt unregistered [ 1214.990825] Key type ._llcrypt unregistered [ 1221.165444] Key type ._llcrypt registered [ 1221.171788] Key type .llcrypt registered [ 1221.368759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1221.386693] alg: No test for adler32 (adler32-zlib) [ 1222.353892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1222.377776] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1229.184846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.131@tcp [ 1229.723871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e remote_dropped -r 1 [ 1229.747396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e remote_dropped -r 1 [ 1229.777589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1229.797797] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1229.987248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1230.018964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1230.046665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1230.076335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1230.102904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1230.139877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1230.174769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1230.205595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.226261] LNet: 62520:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1230.246822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1230.270635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.306850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.337796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.362463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.390858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1230.431852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1230.915994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e remote_error -r 1 [ 1230.941578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e remote_error -r 1 [ 1230.966743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1230.979894] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1231.214898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1231.250433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1231.276236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1231.307891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1231.337772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1231.389778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1231.451451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1231.512900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1231.536340] LNet: 62629:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1231.557708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1231.587880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1231.618931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1231.661276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1231.696589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1231.722374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1231.746771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1232.224901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e remote_timeout -r 1 [ 1232.267210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e remote_timeout -r 1 [ 1232.291509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1232.308611] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1232.488334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1232.524195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1232.569688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1232.606052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1232.648527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1232.679039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1232.720473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1232.760362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1232.779800] LNet: 62738:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1232.801964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1232.845357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1232.880261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1232.912288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1232.945213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1232.969549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1232.995516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1233.359690] LNet: 62784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1233.365833] LNetError: 62784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1233.386257] LNet: Removed LNI 192.168.204.31@tcp [ 1233.714595] Key type .llcrypt unregistered [ 1233.716267] Key type ._llcrypt unregistered [ 1250.405855] Key type ._llcrypt registered [ 1250.408461] Key type .llcrypt registered [ 1251.836948] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 22:37:35 (1787366255) [ 1252.741639] Key type .llcrypt unregistered [ 1252.747821] Key type ._llcrypt unregistered [ 1258.287183] Key type ._llcrypt registered [ 1258.294684] Key type .llcrypt registered [ 1258.580824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1258.613359] alg: No test for adler32 (adler32-zlib) [ 1259.565991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1259.602601] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1268.029493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.131@tcp [ 1277.662764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1277.745738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1277.784037] LNet: Added LNI 192.168.204.31@tcp1 [8/256/0/180] [ 1279.421598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e remote_dropped -r 1 [ 1279.488409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e remote_dropped -r 1 [ 1279.549916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e remote_dropped -r 1 [ 1279.594916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e remote_dropped -r 1 [ 1279.639132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e remote_dropped -r 1 [ 1279.696846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e remote_dropped -r 1 [ 1279.762429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e remote_dropped -r 1 [ 1279.846287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e remote_dropped -r 1 [ 1279.924842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1279.971922] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1279.979804] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.131@tcp [ 1280.211868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1280.246846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1280.279254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1280.310870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.346445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.374676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.409604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.462126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.490755] LNet: 64254:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.532753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.578583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.621421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.653666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.718909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.764301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.810846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1280.840252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1281.408538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e remote_error -r 1 [ 1281.454035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e remote_error -r 1 [ 1281.506453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e remote_error -r 1 [ 1281.535763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e remote_error -r 1 [ 1281.600844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e remote_error -r 1 [ 1281.710450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e remote_error -r 1 [ 1281.762461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e remote_error -r 1 [ 1281.842470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e remote_error -r 1 [ 1281.926328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1281.951883] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1281.960333] LNet: Skipped 2 previous similar messages [ 1282.226276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1282.259538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1282.305562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1282.357398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1282.434638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1282.471456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1282.501584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1282.533155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1282.546695] LNet: 64377:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1282.561838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1282.586884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1282.614463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1282.639406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1282.666221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1282.692927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1282.727660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1282.758263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1283.355350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e remote_timeout -r 1 [ 1283.384626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e remote_timeout -r 1 [ 1283.430898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e remote_timeout -r 1 [ 1283.504783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e remote_timeout -r 1 [ 1283.548334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e remote_timeout -r 1 [ 1283.614438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e remote_timeout -r 1 [ 1283.680507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e remote_timeout -r 1 [ 1283.738362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e remote_timeout -r 1 [ 1283.771181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.131@tcp [ 1283.786687] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1283.792127] LNet: Skipped 1 previous similar message [ 1284.001516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1284.027517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1284.050250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1284.076702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1284.112627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1284.140154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1284.180870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1284.220364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1284.246614] LNet: 64500:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1284.271752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1284.313780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1284.369615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1284.414148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1284.476276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1284.508708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1284.566622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1284.609267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.334901] LNet: 64582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.339907] LNetError: 64582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1290.344718] LNet: Removed LNI 192.168.204.31@tcp [ 1290.358991] LNet: Removed LNI 192.168.204.31@tcp1 [ 1290.842504] Key type .llcrypt unregistered [ 1290.844825] Key type ._llcrypt unregistered [ 1308.922591] Key type ._llcrypt registered [ 1308.925890] Key type .llcrypt registered [ 1310.196723] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 22:38:34 (1787366314) [ 1311.964740] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1313.619326] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 22:38:37 (1787366317) [ 1314.641312] Key type .llcrypt unregistered [ 1314.642957] Key type ._llcrypt unregistered [ 1322.146705] Key type ._llcrypt registered [ 1322.149697] Key type .llcrypt registered [ 1322.369674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1322.386632] alg: No test for adler32 (adler32-zlib) [ 1323.304635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1323.323869] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1330.941135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.131@tcp [ 1331.667838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e network_timeout -r 1 [ 1331.714850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e network_timeout -r 1 [ 1331.786375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.131@tcp [ 1331.813795] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1332.141564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1332.178754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1332.226329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1332.258844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1332.300793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1332.353816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1332.394120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1332.447679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1332.467804] LNet: 66131:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1332.485782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1332.530808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1332.576298] LNet: 1 local NIs in recovery (showing 1): 192.168.204.31@tcp [ 1332.584349] LNet: There was an unexpected network error while writing to 192.168.204.31: rc = -22 [ 1332.591830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1332.623385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1332.666195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1332.699794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1333.011064] LNet: 66174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1333.029762] LNetError: 66174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1334.053940] LNet: Removed LNI 192.168.204.31@tcp [ 1334.453978] Key type .llcrypt unregistered [ 1334.458316] Key type ._llcrypt unregistered [ 1356.018467] Key type ._llcrypt registered [ 1356.021972] Key type .llcrypt registered [ 1356.206706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1356.225810] alg: No test for adler32 (adler32-zlib) [ 1357.188148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1357.230039] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1364.794950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.131@tcp [ 1375.917086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1375.973047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1375.997183] LNet: Added LNI 192.168.204.31@tcp1 [8/256/0/180] [ 1377.083659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e network_timeout -r 1 [ 1377.110615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e network_timeout -r 1 [ 1377.131737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e network_timeout -r 1 [ 1377.161776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e network_timeout -r 1 [ 1377.183408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e network_timeout -r 1 [ 1377.204573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e network_timeout -r 1 [ 1377.228196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e network_timeout -r 1 [ 1377.256253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e network_timeout -r 1 [ 1377.280533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.131@tcp [ 1377.297910] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1377.491884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1377.515404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1377.565737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1377.608871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1377.668940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1377.701909] LNet: 1 local NIs in recovery (showing 1): 192.168.204.31@tcp [ 1377.710802] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.131@tcp [ 1377.719708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1377.753615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1377.790651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1377.815859] LNet: 67057:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1377.847817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1377.902983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1377.962749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1378.014658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1378.060311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1378.089398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1383.805674] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1384.138309] LNet: 67135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.143744] LNetError: 67135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1384.151715] LNet: Removed LNI 192.168.204.31@tcp [ 1384.163164] LNet: Removed LNI 192.168.204.31@tcp1 [ 1384.664838] Key type .llcrypt unregistered [ 1384.668628] Key type ._llcrypt unregistered [ 1407.991903] Key type ._llcrypt registered [ 1407.997422] Key type .llcrypt registered [ 1409.431568] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 22:40:13 (1787366413) [ 1410.256563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1410.267922] alg: No test for adler32 (adler32-zlib) [ 1411.192184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.259167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1411.286132] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1412.112085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1412.175791] LNet: Added LNI 192.168.204.31@tcp1 [8/256/0/180] [ 1413.072878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 1413.126497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1413.159885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1413.185825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1413.208478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1413.236152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1413.259549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.31@tcp [ 1413.279277] LNetError: 68237:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.31@tcp to recovery [ 1413.285966] LNet: 1 local NIs in recovery (showing 1): 192.168.204.31@tcp [ 1413.290429] LNet: There was an unexpected network error while writing to 192.168.204.31: rc = -22 [ 1415.328928] LNet: There was an unexpected network error while writing to 192.168.204.31: rc = -22 [ 1415.756382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1419.424464] LNet: There was an unexpected network error while writing to 192.168.204.31: rc = -22 [ 1420.337384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1420.366658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1420.438848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1420.474213] LNet: 68344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.479044] LNetError: 68344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.483569] LNet: Removed LNI 192.168.204.31@tcp [ 1420.501818] LNet: Removed LNI 192.168.204.31@tcp1 [ 1420.522562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.568924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1420.585685] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1421.317065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1421.338051] LNet: Added LNI 192.168.204.31@tcp1 [8/256/0/180] [ 1422.016022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 1422.046918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1422.095622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1422.134200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1422.161929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1422.199780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1422.234599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1422.271714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.31@tcp [ 1422.289929] LNetError: 68617:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.31@tcp to recovery [ 1422.629448] LNet: There was an unexpected network error while writing to 192.168.204.31: rc = -22 [ 1425.759357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1428.769191] LNet: There was an unexpected network error while writing to 192.168.204.31: rc = -22 [ 1428.774838] LNet: Skipped 1 previous similar message [ 1433.754858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1433.788422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1433.818517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1440.070520] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 22:40:44 (1787366444) [ 1440.310195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1440.333503] LNet: 69335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.348303] LNetError: 69335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1440.363263] LNetError: 69335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1440.375348] LNet: Removed LNI 192.168.204.31@tcp [ 1440.396995] LNet: Removed LNI 192.168.204.31@tcp1 [ 1440.455063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1440.548770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1440.591222] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1441.257728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1441.283617] LNet: Added LNI 192.168.204.31@tcp1 [8/256/0/180] [ 1441.994908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 1442.047278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1442.072684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1442.102743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1442.131378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1442.168442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1442.217339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.31@tcp --health 0 [ 1442.319418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1442.528821] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.31@tcp [ 1443.552706] LNet: There was an unexpected network error while writing to 192.168.204.31: rc = -22 [ 1443.565380] LNet: Skipped 1 previous similar message [ 1458.623755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1458.736366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1458.767155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1458.797358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.31@tcp --health 500 [ 1458.874409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1464.345884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1464.398820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1464.435911] LNet: 69827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.456792] LNetError: 69827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.464895] LNetError: 69827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1464.474720] LNet: Removed LNI 192.168.204.31@tcp [ 1464.490253] LNet: Removed LNI 192.168.204.31@tcp1 [ 1464.515921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.580586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1464.607715] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1465.179514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1465.198168] LNet: Added LNI 192.168.204.31@tcp1 [8/256/0/180] [ 1465.869876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 1465.907945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1465.970444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1466.013721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1466.054453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1466.100610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1466.125256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1466.154190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.31@tcp --health 0 [ 1467.616515] LNet: There was an unexpected network error while writing to 192.168.204.31: rc = -22 [ 1467.626033] LNet: Skipped 3 previous similar messages [ 1468.518351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1478.832271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1478.872922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1478.926746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1485.620765] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 22:41:29 (1787366489) [ 1486.266411] LNet: 70883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1486.275267] LNetError: 70883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.280846] LNetError: 70883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1486.287005] LNet: Removed LNI 192.168.204.31@tcp [ 1486.297150] LNet: Removed LNI 192.168.204.31@tcp1 [ 1486.887669] Key type .llcrypt unregistered [ 1486.893599] Key type ._llcrypt unregistered [ 1492.406821] Key type ._llcrypt registered [ 1492.409654] Key type .llcrypt registered [ 1492.543943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1492.553921] alg: No test for adler32 (adler32-zlib) [ 1493.500702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1493.539076] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1502.798478] Lustre: DEBUG MARKER: Initial discovery [ 1503.068219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.131@tcp [ 1505.538756] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1505.766193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1505.821247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.131@tcp [ 1505.848219] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1505.895464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1505.944729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.131@tcp -r 1 -m GET -l 3 [ 1505.975363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.131@tcp -r 1 -m GET -e local_error [ 1506.004647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.131@tcp -r 1 -m PUT -l 6 [ 1506.850136] LNet: 1 local NIs in recovery (showing 1): 192.168.204.31@tcp [ 1507.475714] Lustre: DEBUG MARKER: Force oleg431-client.virtnet to discover 192.168.204.131@tcp (in background) [ 1507.537828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.131@tcp [ 1509.057589] Lustre: DEBUG MARKER: Force oleg431-server to discover 192.168.204.31@tcp [ 1510.627682] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1511.836808] Lustre: DEBUG MARKER: Wait for 71642 [ 1513.332963] Lustre: DEBUG MARKER: Finished wait on 71642 [ 1513.468331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1513.523979] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1513.527814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1513.820515] LNet: 72103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.827814] LNetError: 72103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1514.881109] LNet: Removed LNI 192.168.204.31@tcp [ 1515.379994] Key type .llcrypt unregistered [ 1515.385589] Key type ._llcrypt unregistered [ 1533.375190] Key type ._llcrypt registered [ 1533.376793] Key type .llcrypt registered [ 1534.629101] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 22:42:18 (1787366538) [ 1535.256296] Key type .llcrypt unregistered [ 1535.258290] Key type ._llcrypt unregistered [ 1541.622429] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1541.772441] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1541.779575] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1541.786329] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1544.313798] Key type ._llcrypt registered [ 1544.318019] Key type .llcrypt registered [ 1544.526851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1544.539914] alg: No test for adler32 (adler32-zlib) [ 1545.497679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.556983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1545.580962] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1546.534762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1546.550503] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1554.521402] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 22:42:38 (1787366558) [ 1554.622214] LNet: 74171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.636752] LNetError: 74171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1554.646520] LNet: Removed LNI 10.1.2.3@tcp [ 1554.661423] LNet: Removed LNI 192.168.204.31@tcp [ 1555.242409] Key type .llcrypt unregistered [ 1555.246706] Key type ._llcrypt unregistered [ 1562.462498] Key type ._llcrypt registered [ 1562.464960] Key type .llcrypt registered [ 1562.712271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1562.739587] alg: No test for adler32 (adler32-zlib) [ 1563.653653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.699961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1563.725057] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1564.453590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1564.477234] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1565.487320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.31@tcp [ 1566.562307] LNetError: Unexpected error -22 connecting to 192.168.204.31@tcp at host 192.168.204.31:988 [ 1571.871924] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 22:42:55 (1787366575) [ 1572.180336] LNet: 75315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.190314] LNetError: 75315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1572.208731] LNet: Removed LNI 10.1.2.3@tcp [ 1572.235847] LNet: Removed LNI 192.168.204.31@tcp [ 1573.097584] Key type .llcrypt unregistered [ 1573.101686] Key type ._llcrypt unregistered [ 1578.996654] Key type ._llcrypt registered [ 1579.000664] Key type .llcrypt registered [ 1579.214362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1579.240949] alg: No test for adler32 (adler32-zlib) [ 1580.194992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.254306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1580.299169] LNet: Added LNI 192.168.204.31@tcp1 [8/256/0/180] [ 1580.959249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1580.978134] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 1581.969853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.31@tcp1 --nid 192.168.204.31@tcp2 [ 1585.971794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.31@tcp1 192.168.204.31@tcp2 [ 1585.994579] LNetError: 76155:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.31@tcp1 to 192.168.204.31@tcp2 - Route table may be misconfigured [ 1586.020826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.31@tcp2 192.168.204.31@tcp1 [ 1592.782103] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 22:43:16 (1787366596) [ 1592.875264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.913488] LNet: 76722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1592.928325] LNetError: 76722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1592.933823] LNet: Removed LNI 192.168.204.31@tcp1 [ 1593.962027] LNet: Removed LNI 192.168.204.31@tcp2 [ 1593.999336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.085582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1594.118370] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1594.852102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1594.883053] LNet: Added LNI 192.168.204.31@tcp1 [8/256/0/180] [ 1595.759582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 1595.815099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.31@tcp 192.168.204.31@tcp [ 1595.877703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.31@tcp1 192.168.204.31@tcp1 [ 1595.928858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.31@tcp -d 192.168.204.31@tcp -e network_timeout [ 1595.969782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -e network_timeout [ 1596.013429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -e network_timeout [ 1596.054967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -e network_timeout [ 1596.103318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 1596.126883] LNet: There was an unexpected network error while writing to 192.168.204.31: rc = -22 [ 1596.146700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1596.276970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1596.390094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1597.088237] LNet: 1 local NIs in recovery (showing 1): 192.168.204.31@tcp [ 1602.585780] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 22:43:26 (1787366606) [ 1602.682802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.715624] LNet: 77580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1602.734315] LNetError: 77580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1602.744384] LNetError: 77580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1602.767422] LNet: Removed LNI 192.168.204.31@tcp [ 1603.817143] LNet: Removed LNI 192.168.204.31@tcp1 [ 1603.855508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.971388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1604.312959] LNet: 77630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.760468] Key type .llcrypt unregistered [ 1604.766093] Key type ._llcrypt unregistered [ 1617.257225] Key type ._llcrypt registered [ 1617.259367] Key type .llcrypt registered [ 1618.490237] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 22:43:42 (1787366622) [ 1618.881602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1618.904284] alg: No test for adler32 (adler32-zlib) [ 1619.859231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.253969] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1622.983740] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 22:43:47 (1787366627) [ 1623.365565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.412251] LNet: 78730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.440910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.521156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1623.555948] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1624.220217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1624.258557] LNet: Added LNI 192.168.204.31@tcp1 [8/256/0/180] [ 1625.259868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 1625.291742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp1 [ 1625.331309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp1 [ 1632.476958] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 22:43:56 (1787366636) [ 1634.105860] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg431-server [ 1635.704610] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 22:43:59 (1787366639) [ 1637.439788] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg431-server [ 1639.137656] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 22:44:03 (1787366643) [ 1640.875700] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg431-server [ 1642.609894] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 22:44:06 (1787366646) [ 1644.311943] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg431-server [ 1646.028393] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 22:44:09 (1787366649) [ 1647.927764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg431-server [ 1649.447501] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 22:44:13 (1787366653) [ 1651.139806] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg431-server [ 1652.725688] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 22:44:16 (1787366656) [ 1654.811221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg431-server [ 1656.613729] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 22:44:20 (1787366660) [ 1658.479930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg431-server [ 1660.307526] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 22:44:24 (1787366664) [ 1662.256886] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg431-server [ 1664.184780] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 22:44:28 (1787366668) [ 1664.430773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.451134] LNet: 82607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1664.456291] LNetError: 82607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.464334] LNet: Removed LNI 192.168.204.31@tcp [ 1665.511533] LNet: Removed LNI 192.168.204.31@tcp1 [ 1665.555216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.647187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1665.673732] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1666.344929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1666.459710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 1666.673903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.697290] LNet: 82754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.705139] LNetError: 82754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1666.717580] LNetError: 82754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1666.737271] LNet: Removed LNI 192.168.204.31@tcp [ 1666.758745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.832244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1666.854546] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1667.624766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 1667.817688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.841951] LNetError: 82900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.858664] LNet: Removed LNI 192.168.204.31@tcp [ 1667.874299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.950605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1667.974857] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1668.540620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1668.669724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 1668.864676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.899903] LNet: 83047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.911866] LNet: 83047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1670.002772] LNet: Removed LNI 192.168.204.31@tcp [ 1670.023727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.102864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1670.131871] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1671.090702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 1671.297708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1671.348889] LNetError: 83192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1671.363385] LNetError: 83192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1671.410191] LNet: Removed LNI 192.168.204.31@tcp [ 1671.443640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1671.489982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1671.514420] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1672.274887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1672.414642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 1672.738733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1673.841929] LNet: Removed LNI 192.168.204.31@tcp [ 1673.899631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1673.970777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1673.997687] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1675.098063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 1675.328402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.373895] LNet: 83485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.388538] LNet: 83485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1675.400500] LNetError: 83485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1675.410620] LNetError: 83485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1675.429152] LNet: Removed LNI 192.168.204.31@tcp [ 1675.450336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.530769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1675.563361] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1676.242424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1676.311138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 1677.722687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.824150] LNet: Removed LNI 192.168.204.31@tcp [ 1678.868617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.960548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1678.991878] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1679.775499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 1680.047415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1681.127838] LNet: Removed LNI 192.168.204.31@tcp [ 1681.168896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.265309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1681.295407] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1682.076862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1682.167554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 1682.444667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.498983] LNet: Removed LNI 192.168.204.31@tcp [ 1683.560608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.636379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1683.663949] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1684.508555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 1685.979066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.011184] LNet: 84088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.023844] LNet: 84088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1686.035308] LNetError: 84088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.043193] LNetError: 84088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1687.083512] LNet: Removed LNI 192.168.204.31@tcp [ 1687.128801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.265893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1687.312688] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1688.097849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1688.184455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 1689.598280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.663346] LNet: Removed LNI 192.168.204.31@tcp [ 1690.712717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.804916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1690.839650] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1691.632448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 1691.862966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1692.901773] LNet: Removed LNI 192.168.204.31@tcp [ 1692.926277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.992459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1693.031305] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1693.766561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1693.828704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1693.857227] LNet: Removed LNI 192.168.204.31@tcp [ 1693.874903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.955734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1693.977385] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1694.779219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1694.859176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.891185] LNet: Removed LNI 192.168.204.31@tcp [ 1694.911123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.957495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1695.036615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.109392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.183903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1695.289873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.326952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.374985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1695.453791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.527188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.589627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1695.624736] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1696.408243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.444827] LNet: Removed LNI 192.168.204.31@tcp [ 1696.460886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.531419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1703.007594] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 22:45:07 (1787366707) [ 1703.250363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.286873] LNet: 85411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.291866] LNet: 85411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1703.327575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.414777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1703.452179] LNet: Added LNI 192.168.204.31@tcp231 [8/256/0/180] [ 1704.803732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.837212] LNetError: 85549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.846140] LNetError: 85549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1704.864614] LNet: Removed LNI 192.168.204.31@tcp231 [ 1704.903099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.997238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1705.064834] LNet: Added LNI 192.168.204.31@tcp231 [8/256/0/180] [ 1705.077559] LNet: 85557:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1705.386781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1705.409629] LNet: Removed LNI 192.168.204.31@tcp231 [ 1705.425985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1705.448746] LNet: Added LNI 192.168.204.31@tcp231 [8/256/0/180] [ 1706.844875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.891835] LNet: Removed LNI 192.168.204.31@tcp231 [ 1706.916277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.989140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1707.030380] LNet: Added LNI 192.168.204.31@tcp231 [8/256/0/0] [ 1707.046155] LNet: 85709:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1707.666215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1707.714154] LNet: Removed LNI 192.168.204.31@tcp231 [ 1707.733272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1707.781496] LNet: Added LNI 192.168.204.31@tcp231 [8/256/0/0] [ 1709.276272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.324095] LNet: Removed LNI 192.168.204.31@tcp231 [ 1709.350273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.423917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1709.459475] LNet: Added LNI 192.168.204.31@tcp231 [8/256/0/60] [ 1709.464796] LNet: 85861:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1709.953729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1710.002795] LNet: Removed LNI 192.168.204.31@tcp231 [ 1710.017264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1710.050662] LNet: Added LNI 192.168.204.31@tcp231 [8/256/0/60] [ 1711.067327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.121974] LNet: Removed LNI 192.168.204.31@tcp231 [ 1711.171043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.280346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1711.332331] LNet: Added LNI 192.168.204.31@tcp231 [8/256/0/180] [ 1711.757681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1711.789824] LNet: Removed LNI 192.168.204.31@tcp231 [ 1711.802902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1711.823673] LNet: Added LNI 192.168.204.31@tcp231 [8/256/0/180] [ 1719.632929] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 22:45:22 (1787366722) [ 1719.750556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.784732] LNet: Removed LNI 192.168.204.31@tcp231 [ 1719.820695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.935222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1719.958210] LNet: Added LNI 192.168.204.31@tcp232 [8/256/0/180] [ 1721.184273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1721.277162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1721.304573] LNet: Removed LNI 192.168.204.31@tcp232 [ 1721.327242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1721.374857] LNet: Added LNI 192.168.204.31@tcp232 [8/256/0/180] [ 1722.489956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1722.548276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1722.591298] LNet: Removed LNI 192.168.204.31@tcp232 [ 1722.616477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1722.639068] LNet: Added LNI 192.168.204.31@tcp232 [8/256/0/180] [ 1723.521450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1723.563368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1723.635330] LNet: Removed LNI 192.168.204.31@tcp232 [ 1723.668565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1723.739098] LNet: Added LNI 192.168.204.31@tcp232 [8/256/0/180] [ 1725.046865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1725.082830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1725.104807] LNet: Removed LNI 192.168.204.31@tcp232 [ 1725.121500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1732.376348] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 22:45:36 (1787366736) [ 1733.836188] Key type .llcrypt unregistered [ 1733.840515] Key type ._llcrypt unregistered [ 1742.340265] Key type ._llcrypt registered [ 1742.344583] Key type .llcrypt registered [ 1742.803422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1742.856508] alg: No test for adler32 (adler32-zlib) [ 1743.908791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1743.986355] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1752.120466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.131@tcp [ 1760.217667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.276791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1760.308971] LNet: Added LNI 192.168.204.31@tcp1 [8/256/0/180] [ 1761.701504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_interrupt -r 2 [ 1761.746419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e local_interrupt -r 2 [ 1761.795527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_interrupt -r 2 [ 1761.823145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e local_interrupt -r 2 [ 1761.856754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e local_interrupt -r 2 [ 1761.884991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e local_interrupt -r 2 [ 1761.914430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e local_interrupt -r 2 [ 1761.941848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e local_interrupt -r 2 [ 1761.977777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.31@tcp 192.168.204.131@tcp [ 1762.000487] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1762.005971] LNet: 1 local NIs in recovery (showing 1): 192.168.204.31@tcp [ 1762.220567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1762.259744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1762.286132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1762.316994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1762.359266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1762.425848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1762.475297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1762.501745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1762.518291] LNet: 88516:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1762.532694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1762.563409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1762.604201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1762.644656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1762.674978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1762.718213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1763.314898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_dropped -r 2 [ 1763.382900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e local_dropped -r 2 [ 1763.421761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_dropped -r 2 [ 1763.474352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e local_dropped -r 2 [ 1763.511449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e local_dropped -r 2 [ 1763.549347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e local_dropped -r 2 [ 1763.582355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e local_dropped -r 2 [ 1763.623862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e local_dropped -r 2 [ 1763.662702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.31@tcp 192.168.204.131@tcp [ 1763.684250] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1763.880934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1763.910898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1763.934300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1763.962387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1763.991741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1764.028304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1764.069917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1764.105232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1764.128187] LNet: 88637:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1764.142796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1764.178067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1764.203691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1764.236528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1764.277667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1764.319974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1764.738812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_aborted -r 2 [ 1764.767578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e local_aborted -r 2 [ 1764.797324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_aborted -r 2 [ 1764.825276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e local_aborted -r 2 [ 1764.864865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e local_aborted -r 2 [ 1764.917198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e local_aborted -r 2 [ 1764.959253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e local_aborted -r 2 [ 1764.999348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e local_aborted -r 2 [ 1765.048361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.31@tcp 192.168.204.131@tcp [ 1765.081774] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1765.270860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1765.295330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1765.329460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1765.360811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1765.389173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1765.416612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1765.438861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1765.464035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1765.483868] LNet: 88757:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1765.512080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1765.549613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1765.579872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1765.602194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1765.630800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1765.665383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1766.058355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_no_route -r 2 [ 1766.096950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e local_no_route -r 2 [ 1766.128590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_no_route -r 2 [ 1766.162649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e local_no_route -r 2 [ 1766.194025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e local_no_route -r 2 [ 1766.224326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e local_no_route -r 2 [ 1766.252911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e local_no_route -r 2 [ 1766.277420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e local_no_route -r 2 [ 1766.306353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.31@tcp 192.168.204.131@tcp [ 1766.500460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1766.526102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1766.566803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1766.603383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1766.632066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1766.662231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1766.701395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1766.726748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1766.747064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1766.777627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1766.805792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1766.840933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1766.878381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1766.905551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1767.369959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e local_timeout -r 2 [ 1767.409733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e local_timeout -r 2 [ 1767.443911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e local_timeout -r 2 [ 1767.476221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e local_timeout -r 2 [ 1767.503700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e local_timeout -r 2 [ 1767.536544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e local_timeout -r 2 [ 1767.577902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e local_timeout -r 2 [ 1767.610638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e local_timeout -r 2 [ 1767.641827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.31@tcp 192.168.204.131@tcp [ 1767.661492] LNet: There was an unexpected network error while writing to 192.168.204.131: rc = -22 [ 1767.669638] LNet: Skipped 1 previous similar message [ 1767.894216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1767.949121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1767.988193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1768.018987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1768.045345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1768.082609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1768.116455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1768.149399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1768.162229] LNet: 88997:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1768.167538] LNet: 88997:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1768.189053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1768.217824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1768.253067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1768.292870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1768.335317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1768.374582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1768.963900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp -m GET -e remote_dropped -r 2 [ 1768.993138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.131@tcp1 -m GET -e remote_dropped -r 2 [ 1769.056933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp -m GET -e remote_dropped -r 2 [ 1769.092231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp -d 192.168.204.31@tcp1 -m GET -e remote_dropped -r 2 [ 1769.128385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp -m GET -e remote_dropped -r 2 [ 1769.157266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.131@tcp1 -m GET -e remote_dropped -r 2 [ 1769.216427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp -m GET -e remote_dropped -r 2 [ 1769.261271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.31@tcp1 -d 192.168.204.31@tcp1 -m GET -e remote_dropped -r 2 [ 1769.292781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.31@tcp 192.168.204.131@tcp [ 1769.311168] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.131@tcp [ 1769.584925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1769.609894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1769.636288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1769.670993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1769.700994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1769.726787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1769.751780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1769.774749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1769.822788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1769.889633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1769.931539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1769.978517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1770.026703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1770.075172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1775.737644] LNet: 89195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.744924] LNetError: 89195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.751846] LNet: Removed LNI 192.168.204.31@tcp [ 1775.763236] LNet: Removed LNI 192.168.204.31@tcp1 [ 1776.260142] Key type .llcrypt unregistered [ 1776.261902] Key type ._llcrypt unregistered [ 1792.129661] Key type ._llcrypt registered [ 1792.133065] Key type .llcrypt registered [ 1793.444235] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 22:46:37 (1787366797) [ 1793.831054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1793.850395] alg: No test for adler32 (adler32-zlib) [ 1794.777984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.830398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1794.857306] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1795.922748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1795.955503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.31@tcp [ 1795.981557] LNetError: 90189:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.31@tcp to recovery [ 1796.085748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.31@tcp [ 1796.226281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1802.018505] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 22:46:45 (1787366805) [ 1802.276292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1802.298391] LNet: 90780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1802.317449] LNetError: 90780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1802.335626] LNet: Removed LNI 192.168.204.31@tcp [ 1802.352110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.405921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1802.441089] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1803.062602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1803.078743] LNet: Added LNI 192.168.204.31@tcp1 [8/256/0/180] [ 1804.201860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.31@tcp [ 1804.225980] LNetError: 91047:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.31@tcp to recovery [ 1804.448525] LNet: 1 local NIs in recovery (showing 1): 192.168.204.31@tcp [ 1804.462053] LNet: There was an unexpected network error while writing to 192.168.204.31: rc = -22 [ 1805.492614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.31@tcp [ 1813.273870] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 22:46:56 (1787366816) [ 1816.007697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg431-server [ 1818.065597] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 22:47:01 (1787366821) [ 1819.975960] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg431-server [ 1822.444774] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 22:47:05 (1787366825) [ 1824.848191] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg431-server [ 1826.781115] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 22:47:10 (1787366830) [ 1826.911874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.939059] LNet: 92667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1826.969226] LNetError: 92667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1826.979089] LNet: Removed LNI 192.168.204.31@tcp [ 1827.003445] LNet: Removed LNI 192.168.204.31@tcp1 [ 1827.039699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.117879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.132512] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1828.066816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1828.205988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1828.256888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1828.311904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1828.352246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1828.406532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1836.102613] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 22:47:19 (1787366839) [ 1836.405288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1836.438173] LNet: 93394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.446926] LNetError: 93394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1836.461312] LNetError: 93394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1836.487568] LNet: Removed LNI 192.168.204.31@tcp [ 1836.506762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.561464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1836.589566] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1843.704738] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 22:47:27 (1787366847) [ 1843.946397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.011387] LNet: 94096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.024277] LNetError: 94096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.057156] LNet: Removed LNI 192.168.204.31@tcp [ 1844.093741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.142572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1844.175751] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1851.528271] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 22:47:35 (1787366855) [ 1853.201467] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1855.054634] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 22:47:38 (1787366858) [ 1855.506785] LNet: 95133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.515472] LNetError: 95133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.539368] LNet: Removed LNI 192.168.204.31@tcp [ 1856.140797] Key type .llcrypt unregistered [ 1856.147667] Key type ._llcrypt unregistered [ 1862.580253] Key type ._llcrypt registered [ 1862.581552] Key type .llcrypt registered [ 1862.771152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1862.788261] alg: No test for adler32 (adler32-zlib) [ 1863.774400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1863.818720] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1871.520207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.131@tcp [ 1876.729533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.131@tcp [ 1877.230364] LNet: 95559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.235781] LNetError: 95559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.254642] LNet: Removed LNI 192.168.204.31@tcp [ 1877.779801] Key type .llcrypt unregistered [ 1877.783734] Key type ._llcrypt unregistered [ 1891.308468] Key type ._llcrypt registered [ 1891.310280] Key type .llcrypt registered [ 1892.601956] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1894.326699] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1896.174184] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 22:48:20 (1787366900) [ 1897.983734] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1899.693424] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 22:48:23 (1787366903) [ 1901.562608] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg431-server [ 1903.200584] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 22:48:27 (1787366907) [ 1905.126107] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg431-server [ 1906.934186] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 22:48:30 (1787366910) [ 1908.569950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1908.604556] alg: No test for adler32 (adler32-zlib) [ 1909.795951] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 1910.102517] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1911.792810] Key type lgssc registered [ 1912.920907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1933.987487] Key type lgssc unregistered [ 1934.202503] LNet: 98738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1934.208733] LNetError: 98738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1934.226824] LNet: Removed LNI 192.168.204.31@tcp [ 1934.978562] Key type .llcrypt unregistered [ 1934.981079] Key type ._llcrypt unregistered [ 1950.131115] Key type ._llcrypt registered [ 1950.133600] Key type .llcrypt registered [ 1951.613739] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 22:49:15 (1787366955) [ 1952.127967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1952.140399] alg: No test for adler32 (adler32-zlib) [ 1953.091556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1953.208582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1953.300177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1953.370703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1953.439744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1961.324738] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 22:49:24 (1787366964) [ 1961.556260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1961.589379] LNet: 100154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1961.613748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1961.685846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1961.709045] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1962.714627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1962.748515] LNet: 100298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.756337] LNetError: 100298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1962.772919] LNet: Removed LNI 192.168.204.31@tcp [ 1962.799921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.855917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1962.892919] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1963.358500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.396815] LNetError: 100327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.418290] LNet: Removed LNI 192.168.204.31@tcp [ 1963.482297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.597957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1963.648599] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1971.929349] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 22:49:35 (1787366975) [ 1972.030393] LNet: 100910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.036767] LNet: 100910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1972.049763] LNetError: 100910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1972.084772] LNet: Removed LNI 192.168.204.31@tcp [ 1972.725782] Key type .llcrypt unregistered [ 1972.730079] Key type ._llcrypt unregistered [ 1979.501292] Key type ._llcrypt registered [ 1979.506505] Key type .llcrypt registered [ 1979.750526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1979.768439] alg: No test for adler32 (adler32-zlib) [ 1980.720914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1980.795492] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 1980.799620] LNet: *** cfs_fail_loc=e003, val=100*** [ 1981.922600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 1981.958597] LNet: *** cfs_fail_loc=e003, val=100*** [ 1981.963612] LNet: Skipped 2 previous similar messages [ 1981.985064] Ping reply received with mapping: [ 1981.986772] found metadata reply NI[192.168.204.31@tcp] [ 1981.991720] 1f cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1981.998116] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1982.009081] 5a 5a 5a 5a 5a 5a 5a 5a [ 1982.263744] LNet: 101319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1982.272514] LNetError: 101319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.294447] LNet: Removed LNI 192.168.204.31@tcp [ 1982.965553] Key type .llcrypt unregistered [ 1982.967365] Key type ._llcrypt unregistered [ 1996.639801] Key type ._llcrypt registered [ 1996.641914] Key type .llcrypt registered [ 1998.368200] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 22:50:01 (1787367001) [ 1998.992457] Key type .llcrypt unregistered [ 1998.995484] Key type ._llcrypt unregistered [ 2006.257389] Key type ._llcrypt registered [ 2006.262492] Key type .llcrypt registered [ 2006.609044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2006.640109] alg: No test for adler32 (adler32-zlib) [ 2007.703119] LNetError: 102377:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2008.068860] Key type .llcrypt unregistered [ 2008.072435] Key type ._llcrypt unregistered [ 2013.573221] Key type ._llcrypt registered [ 2013.575128] Key type .llcrypt registered [ 2014.684267] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 22:50:18 (1787367018) [ 2015.500488] Key type .llcrypt unregistered [ 2015.507972] Key type ._llcrypt unregistered [ 2021.308907] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2021.311296] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2024.024953] Key type ._llcrypt registered [ 2024.028562] Key type .llcrypt registered [ 2024.352822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2024.374806] alg: No test for adler32 (adler32-zlib) [ 2025.374148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2025.493193] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2035.333711] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 22:50:39 (1787367039) [ 2035.676870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2035.711269] LNet: 104166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.719814] LNetError: 104166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2035.752503] LNet: Removed LNI 10.1.2.3@tcp [ 2035.784559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2036.649596] Key type .llcrypt unregistered [ 2036.656799] Key type ._llcrypt unregistered [ 2043.611032] Key type ._llcrypt registered [ 2043.612441] Key type .llcrypt registered [ 2043.831664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2043.852887] alg: No test for adler32 (adler32-zlib) [ 2044.807549] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2044.861958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 2045.013750] LNet: 104458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.023333] LNetError: 104458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.054786] LNet: Removed LNI 192.168.204.31@tcp [ 2046.815407] Key type .llcrypt unregistered [ 2046.817309] Key type ._llcrypt unregistered [ 2059.207528] Key type ._llcrypt registered [ 2059.212839] Key type .llcrypt registered [ 2060.578289] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 22:51:04 (1787367064) [ 2061.653143] Key type .llcrypt unregistered [ 2061.654587] Key type ._llcrypt unregistered [ 2069.103698] Key type ._llcrypt registered [ 2069.106194] Key type .llcrypt registered [ 2069.430498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2069.461150] alg: No test for adler32 (adler32-zlib) [ 2076.985194] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 22:51:21 (1787367081) [ 2077.407853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2077.516031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2077.636243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2077.691598] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2078.380887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2078.429154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2078.470963] LNet: Removed LNI 192.168.204.31@tcp [ 2078.730306] LNet: 106214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.181631] Key type .llcrypt unregistered [ 2079.183830] Key type ._llcrypt unregistered [ 2092.997140] Key type ._llcrypt registered [ 2093.002761] Key type .llcrypt registered [ 2095.072862] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 22:51:38 (1787367098) [ 2095.567118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2095.611242] alg: No test for adler32 (adler32-zlib) [ 2096.636244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.691948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2096.716714] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2097.787691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 2097.829297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.31@tcp [ 2097.849780] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2097.854285] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2097.856275] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2097, 0, 0 [ 2097.860915] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2097.863224] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2097.866377] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2097.869322] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2097, 0, 0 [ 2097.874761] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2097.877275] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2097.879968] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2097.885386] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2097, 0, 2147 [ 2097.889349] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2097.891269] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2097.895859] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2097.899380] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2097, 0, 2147 [ 2097.904127] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2104.938555] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 22:51:48 (1787367108) [ 2105.401161] LNet: 107754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.408708] LNetError: 107754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2105.436754] LNet: Removed LNI 192.168.204.31@tcp [ 2106.409580] Key type .llcrypt unregistered [ 2106.412985] Key type ._llcrypt unregistered [ 2113.928400] Key type ._llcrypt registered [ 2113.931971] Key type .llcrypt registered [ 2114.267111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2114.284837] alg: No test for adler32 (adler32-zlib) [ 2115.260329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2115.309516] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2123.724825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.131@tcp [ 2134.409426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.514602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2134.563809] LNet: Added LNI 192.168.204.31@tcp1 [8/256/0/180] [ 2135.797488] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2135.919660] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2135.923731] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2135.927969] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2138.304652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2138.346167] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2149.784637] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 22:52:33 (1787367153) [ 2150.106102] LNet: 109267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.113227] LNetError: 109267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2150.121746] LNet: Removed LNI 192.168.204.31@tcp [ 2151.136210] LNet: Removed LNI 192.168.204.31@tcp1 [ 2151.140160] LNetError: 109267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2151.148501] LNetError: 109267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2151.190356] LNet: Removed LNI 10.1.2.3@tcp99 [ 2151.736891] Key type .llcrypt unregistered [ 2151.740648] Key type ._llcrypt unregistered [ 2159.708894] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2159.712221] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2162.296901] Key type ._llcrypt registered [ 2162.300259] Key type .llcrypt registered [ 2162.556312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2162.578980] alg: No test for adler32 (adler32-zlib) [ 2163.550439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2163.605939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2163.653565] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2164.392464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2164.421352] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2166.918416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2166.991350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2167.061636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2167.095335] LNetError: 110015:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2167.148138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2167.204950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2167.671635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2167.745255] LNet: 110035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2167.754462] LNetError: 110035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2167.762276] LNet: Removed LNI 10.1.2.3@tcp [ 2168.819378] LNet: Removed LNI 192.168.204.31@tcp [ 2169.494475] Key type .llcrypt unregistered [ 2169.496514] Key type ._llcrypt unregistered [ 2181.697441] Key type ._llcrypt registered [ 2181.701609] Key type .llcrypt registered [ 2183.218826] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 22:53:06 (1787367186) [ 2183.642260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2183.655263] alg: No test for adler32 (adler32-zlib) [ 2184.666248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2184.708114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2184.728224] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2185.538976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 2185.668919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg431-client@tcp [ 2191.766475] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 22:53:15 (1787367195) [ 2191.976084] LNet: 111568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.985544] LNetError: 111568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2192.020287] LNet: Removed LNI 192.168.204.31@tcp [ 2192.668779] Key type .llcrypt unregistered [ 2192.673037] Key type ._llcrypt unregistered [ 2202.170279] Key type ._llcrypt registered [ 2202.176130] Key type .llcrypt registered [ 2202.331785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2202.340228] alg: No test for adler32 (adler32-zlib) [ 2203.292676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2203.416837] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2203.516302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2214.028430] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 22:53:37 (1787367217) [ 2215.720586] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2218.150436] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 22:53:41 (1787367221) [ 2219.992199] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2222.287610] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 22:53:45 (1787367225) [ 2222.451523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2222.485438] LNet: 112992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.494773] LNetError: 112992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.516562] LNet: Removed LNI 192.168.204.31@tcp [ 2222.558193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.672041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2222.700231] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2223.513807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2223.571330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2231.835479] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 22:53:55 (1787367235) [ 2231.930585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2231.981347] LNet: 113702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2232.000603] LNetError: 113702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2232.041617] LNet: Removed LNI 192.168.204.31@tcp [ 2232.107940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.176801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2232.212460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2232.969357] Key type .llcrypt unregistered [ 2232.974354] Key type ._llcrypt unregistered [ 2245.193184] Key type ._llcrypt registered [ 2245.194573] Key type .llcrypt registered [ 2246.119322] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 22:54:10 (1787367250) [ 2246.977973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2246.997919] alg: No test for adler32 (adler32-zlib) [ 2248.135331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.259143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2248.363652] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2249.354749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2249.445377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2255.542979] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 22:54:19 (1787367259) [ 2255.818249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2255.829764] LNet: 115268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2255.833708] LNetError: 115268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2255.846386] LNet: Removed LNI 192.168.204.31@tcp [ 2255.862245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2255.893791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2255.913693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2262.423496] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 22:54:26 (1787367266) [ 2262.686037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2262.701967] LNet: 115848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2262.730641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2262.828080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2263.107389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2263.142264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2263.217923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2263.548900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2263.650587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2263.792533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2263.851062] LNet: 115881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.886218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.997565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2264.455150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2264.490659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2264.544666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2264.816371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2264.891370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2271.102295] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 22:54:35 (1787367275) [ 2271.323823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2271.337901] LNet: 116482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.354399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.402725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2271.425239] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2272.128863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2272.151556] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 2273.093605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2273.195531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2273.513950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.536890] LNetError: 116761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.543310] LNet: Removed LNI 192.168.204.31@tcp [ 2273.554294] LNet: Removed LNI 192.168.204.31@tcp2 [ 2273.589476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.656840] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2273.665482] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 2273.680272] LNet: 116767:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2274.276779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2274.579578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2274.635576] LNetError: 116804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2274.647406] LNetError: 116804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2274.662891] LNet: Removed LNI 192.168.204.31@tcp [ 2274.677613] LNet: Removed LNI 192.168.204.31@tcp2 [ 2274.694726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2274.780513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2274.812431] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2275.607705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2275.624638] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 2276.464776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2276.523844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2276.601359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2276.883361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2276.903588] LNet: 117089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.919227] LNet: 117089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2276.928646] LNetError: 117089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.939817] LNetError: 117089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2277.985700] LNet: Removed LNI 192.168.204.31@tcp [ 2279.017822] LNet: Removed LNI 192.168.204.31@tcp2 [ 2279.050203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2279.145210] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2279.157985] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 2279.169669] LNet: 117095:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2279.868439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2280.146719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2280.167379] LNet: Removed LNI 192.168.204.31@tcp [ 2281.190923] LNet: Removed LNI 192.168.204.31@tcp2 [ 2281.231193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2281.311582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2281.345123] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2282.154646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2282.176231] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 2283.294559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2283.398593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2283.585733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2283.610931] LNetError: 117411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2283.617024] LNetError: 117411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2283.623962] LNet: Removed LNI 192.168.204.31@tcp [ 2283.645718] LNet: Removed LNI 192.168.204.31@tcp2 [ 2283.662814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2283.710529] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2283.716748] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 2283.724827] LNet: 117417:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2284.144953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2284.343447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2285.408179] LNet: Removed LNI 192.168.204.31@tcp [ 2285.416761] LNet: Removed LNI 192.168.204.31@tcp2 [ 2285.490060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2285.549922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2285.587041] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2286.457325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2286.488113] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 2287.459924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2287.515265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2287.583199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2287.756827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2287.772381] LNet: 117739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2287.777925] LNet: 117739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2288.801431] LNet: Removed LNI 192.168.204.31@tcp [ 2289.835057] LNet: Removed LNI 192.168.204.31@tcp2 [ 2289.885494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2289.962468] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2289.971329] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 2289.981207] LNet: 117745:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2290.419825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2290.582203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2290.599368] LNet: Removed LNI 192.168.204.31@tcp [ 2291.631357] LNet: Removed LNI 192.168.204.31@tcp2 [ 2291.662589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2291.739198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2291.796637] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2292.395896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2292.418132] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 2293.249279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2293.281246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2293.353858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2293.709826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2293.731053] LNetError: 118079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2293.739874] LNetError: 118079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2293.744895] LNet: Removed LNI 192.168.204.31@tcp [ 2293.774331] LNet: Removed LNI 192.168.204.31@tcp2 [ 2293.790955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.840758] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2293.851483] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 2294.332559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2294.629513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2295.712333] LNet: Removed LNI 192.168.204.31@tcp [ 2295.727380] LNet: Removed LNI 192.168.204.31@tcp2 [ 2295.773623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.834438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2295.863978] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2296.501559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2296.516857] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 2297.431638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2297.469507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2297.495958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2297.565306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2297.941194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2298.976265] LNet: Removed LNI 192.168.204.31@tcp [ 2300.013250] LNet: Removed LNI 192.168.204.31@tcp2 [ 2300.056605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2300.108551] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2300.114601] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 2300.119530] LNet: 118445:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2300.126915] LNet: 118445:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 2300.673946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2307.424204] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2308.953354] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 22:55:12 (1787367312) [ 2309.142942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2309.161431] LNet: 119204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.165657] LNet: 119204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2309.173393] LNet: Removed LNI 192.168.204.31@tcp [ 2310.252838] LNet: Removed LNI 192.168.204.31@tcp2 [ 2310.305374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.646078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2310.680522] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2311.388910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2311.407781] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2312.462282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2312.492361] LNetError: 119478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2312.500995] LNetError: 119478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2312.506980] LNet: Removed LNI 10.1.2.3@tcp [ 2312.530501] LNet: Removed LNI 192.168.204.31@tcp [ 2312.553781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.659182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2312.686525] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2312.689522] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2313.203310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2313.236248] LNet: Removed LNI 192.168.204.31@tcp [ 2313.256517] LNet: Removed LNI 10.1.2.3@tcp [ 2319.427983] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 22:55:23 (1787367323) [ 2319.527765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2319.559551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.616946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2319.640374] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2320.327919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2320.374491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 2320.416481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 2320.449238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 2320.477225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 2320.502571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 2320.546592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 2320.608321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 2320.646975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 2320.684300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 2320.745568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.31@tcp [ 2320.888743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2327.607241] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 22:55:31 (1787367331) [ 2327.866860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2328.937184] LNet: Removed LNI 192.168.204.31@tcp [ 2328.966134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2329.115470] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2329.293246] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2329.302944] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2329.309428] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2331.662471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2331.686435] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2332.386366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2332.409895] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2334.308484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2334.340387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2334.378820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.31@tcp -m PUT -e local_timeout -r 1 [ 2334.424947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2334.478640] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2334.482485] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2334.668568] LNetError: 120885:0:(lib-move.c:4801:lnet_parse()) 192.168.204.31@tcp, src 192.168.204.31@tcp: Dropping PUT (error -108 looking up sender) [ 2335.715763] LNet: Removed LNI 10.1.2.3@tcp [ 2336.745216] LNet: Removed LNI 192.168.204.31@tcp [ 2337.288453] Key type .llcrypt unregistered [ 2337.291831] Key type ._llcrypt unregistered [ 2350.183149] Key type ._llcrypt registered [ 2350.185069] Key type .llcrypt registered [ 2351.771920] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 22:55:55 (1787367355) [ 2352.155086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2352.172185] alg: No test for adler32 (adler32-zlib) [ 2353.015936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.430562] LNet: 122200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.536371] Key type .llcrypt unregistered [ 2354.544242] Key type ._llcrypt unregistered [ 2363.428766] Key type ._llcrypt registered [ 2363.432527] Key type .llcrypt registered [ 2363.667633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2363.680533] alg: No test for adler32 (adler32-zlib) [ 2364.620910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2364.660648] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2373.051377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.131@tcp [ 2373.313257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2373.798159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.131@tcp [ 2373.920781] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.131@tcp [ 2374.064927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2374.094990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2374.116428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2374.141468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2374.170484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2374.192898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2374.223628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2374.275391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2374.304359] LNet: 122654:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2374.320353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2374.346622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2374.380676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2374.408384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2374.432860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2374.463899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2375.023551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2375.228765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2375.263717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2375.308839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2375.374785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2375.407881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2375.452345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2375.519321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2375.602139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2375.622899] LNet: 122771:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2375.650160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2375.716387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2375.763948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2375.819278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2375.871698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2375.940757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2376.669672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.131@tcp [ 2376.881503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2376.907688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2376.932138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2376.958889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2376.987471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2377.015353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2377.043767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2377.084846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2377.117128] LNet: 122888:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2377.154481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2377.218727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2377.263477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2377.305309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2377.332343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2377.358802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2377.865447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2378.058604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2378.104673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2378.154312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2378.189972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2378.247166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2378.281592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2378.322300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2378.363355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2378.403271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2378.439430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2378.497320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2378.565632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2378.624357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2378.663888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2379.146264] LNet: 123056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2379.154643] LNetError: 123056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2379.172885] LNet: Removed LNI 192.168.204.31@tcp [ 2379.745845] Key type .llcrypt unregistered [ 2379.757308] Key type ._llcrypt unregistered [ 2399.505720] Key type ._llcrypt registered [ 2399.509819] Key type .llcrypt registered [ 2400.878403] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 22:56:44 (1787367404) [ 2401.496513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2401.519902] alg: No test for adler32 (adler32-zlib) [ 2402.462741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2402.817759] LNet: 123943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2403.540727] Key type .llcrypt unregistered [ 2403.546505] Key type ._llcrypt unregistered [ 2410.648633] Key type ._llcrypt registered [ 2410.651763] Key type .llcrypt registered [ 2410.864501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2410.892479] alg: No test for adler32 (adler32-zlib) [ 2411.822632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2411.857975] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2420.733637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.131@tcp [ 2430.922612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2430.987954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2431.024985] LNet: Added LNI 192.168.204.31@tcp1 [8/256/0/180] [ 2432.188133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2432.986587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.131@tcp [ 2433.344941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2433.380279] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.131@tcp [ 2433.392579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2433.431434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2433.478192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2433.501403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2433.543144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2433.579992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2433.621746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2433.641790] LNet: 124598:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2433.663701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2433.707814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2433.759550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2433.830046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2433.875857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2433.926219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2434.662465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2434.926256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2434.960552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2434.991812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2435.026744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2435.056490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2435.081169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2435.108634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2435.133561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2435.151892] LNet: 124716:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2435.170274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2435.232868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2435.285166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2435.316987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2435.360810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2435.394994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2436.059885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.131@tcp [ 2436.309300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2436.339577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2436.400760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2436.438585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2436.470992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2436.507290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2436.538907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2436.579423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2436.603188] LNet: 124833:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2436.623489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2436.676599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2436.720948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2436.754293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2436.795931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2436.829611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2437.472409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2437.762258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2437.809236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2437.858567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2437.918260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2437.982652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2438.025884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2438.059222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2438.103113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2438.139632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2438.203499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2438.241502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2438.275952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2438.320300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2438.362680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2443.204956] LNet: 125035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2443.211907] LNetError: 125035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2443.218973] LNet: Removed LNI 192.168.204.31@tcp [ 2443.238825] LNet: Removed LNI 192.168.204.31@tcp1 [ 2443.694136] Key type .llcrypt unregistered [ 2443.699853] Key type ._llcrypt unregistered [ 2458.765663] Key type ._llcrypt registered [ 2458.768135] Key type .llcrypt registered [ 2460.060541] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 22:57:43 (1787367463) [ 2460.467559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2460.481614] alg: No test for adler32 (adler32-zlib) [ 2461.398511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2461.465132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2461.481642] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2462.159849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2462.188959] LNet: Added LNI 192.168.204.31@tcp2 [8/256/0/180] [ 2462.957578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.31@tcp [ 2463.024419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2464.096324] LNet: Removed LNI 192.168.204.31@tcp [ 2470.765353] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 22:57:54 (1787367474) [ 2472.373972] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2474.706545] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 22:57:58 (1787367478) [ 2482.039880] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 22:58:06 (1787367486) [ 2489.019038] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 22:58:13 (1787367493) [ 2497.584488] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 22:58:21 (1787367501) [ 2504.819349] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 22:58:28 (1787367508) [ 2513.561743] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 22:58:37 (1787367517) [ 2521.127245] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 22:58:45 (1787367525) [ 2528.419706] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 22:58:52 (1787367532) [ 2537.113184] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 22:59:00 (1787367540) [ 2543.495573] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 22:59:07 (1787367547) [ 2550.670885] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 22:59:14 (1787367554) [ 2558.789173] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 22:59:22 (1787367562) [ 2565.984779] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 22:59:29 (1787367569) [ 2575.169280] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 22:59:38 (1787367578) [ 2583.260475] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 22:59:47 (1787367587) [ 2591.358463] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 22:59:54 (1787367594) [ 2599.438945] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 23:00:03 (1787367603) [ 2607.678598] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 23:00:11 (1787367611) [ 2607.883916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2607.934099] LNet: 136728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2607.950656] LNetError: 136728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2607.970513] LNet: Removed LNI 192.168.204.31@tcp2 [ 2608.012143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2608.232689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2608.278614] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2609.914742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2609.932609] LNet: 136871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2609.938962] LNetError: 136871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2609.955604] LNet: Removed LNI 192.168.204.31@tcp [ 2609.975562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2610.036032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2610.071241] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2617.833766] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 23:00:21 (1787367621) [ 2626.567646] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 23:00:30 (1787367630) [ 2634.759289] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 23:00:38 (1787367638) [ 2643.175476] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 23:00:46 (1787367646) [ 2650.594878] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 23:00:54 (1787367654) [ 2658.675790] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 23:01:02 (1787367662) [ 2666.869582] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 23:01:10 (1787367670) [ 2674.745585] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 23:01:18 (1787367678) [ 2683.702894] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 23:01:26 (1787367686) [ 2695.520707] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 23:01:38 (1787367698) [ 2704.044904] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 23:01:47 (1787367707) [ 2713.366813] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 23:01:56 (1787367716) [ 2721.672112] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2458 sec ========== 23:02:04 (1787367724) [ 2722.057546] LNet: 144351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2722.067877] LNetError: 144351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2722.091611] LNet: Removed LNI 192.168.204.31@tcp [ 2722.858497] Key type .llcrypt unregistered [ 2722.872871] Key type ._llcrypt unregistered [ 2735.346914] Key type ._llcrypt registered [ 2735.352347] Key type .llcrypt registered [ 2735.778739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2735.796852] alg: No test for adler32 (adler32-zlib) [ 2736.841505] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 2737.083541] LNet: Added LNI 192.168.204.31@tcp [8/256/0/180] [ 2738.792225] Key type lgssc registered [ 2740.143247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2803.088526] Lustre: Mounted lustre-client - version 2.17.57_46_g5381a3b [ 2808.298618] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2816.396792] Lustre: Unmounted lustre-client [ 2845.702798] Key type lgssc unregistered [ 2845.997612] LNet: 147326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2846.007348] LNetError: 147326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2846.029493] LNet: Removed LNI 192.168.204.31@tcp [ 2846.857406] Key type .llcrypt unregistered [ 2846.862419] Key type ._llcrypt unregistered