[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 443009077 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003190] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008427] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011138] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013038] SELinux: Initializing. [ 0.014082] *** VALIDATE selinux *** [ 0.022808] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027177] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028214] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030126] *** VALIDATE tmpfs *** [ 0.032139] *** VALIDATE proc *** [ 0.033301] *** VALIDATE cgroup *** [ 0.034014] *** VALIDATE cgroup2 *** [ 0.035253] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039011] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.042759] debug: unmapping init [mem 0xffffffff8a459000-0xffffffff8a460fff] [ 0.044179] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045735] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051019] ... fixed-purpose events: 3 [ 0.052030] ... event mask: 000000070000000f [ 0.053339] rcu: Hierarchical SRCU implementation. [ 0.055530] smp: Bringing up secondary CPUs ... [ 0.056623] x86: Booting SMP configuration: [ 0.057030] .... node #0, CPUs: #1 #2 #3 [ 0.060653] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138903] node 0 deferred pages initialised in 73ms [ 0.142105] devtmpfs: initialized [ 0.144032] x86/mm: Memory block size: 128MB [ 0.146757] gcov: version magic: 0x41383552 [ 0.148258] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151273] pinctrl core: initialized pinctrl subsystem [ 0.152219] [ 0.152864] ************************************************************* [ 0.153013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154014] ** ** [ 0.155021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156015] ** ** [ 0.157014] ** This means that this kernel is built to expose internal ** [ 0.158018] ** IOMMU data structures, which may compromise security on ** [ 0.159013] ** your system. ** [ 0.160019] ** ** [ 0.161017] ** If you see this message and you are not debugging the ** [ 0.162018] ** kernel, report this immediately to your vendor! ** [ 0.163013] ** ** [ 0.164010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165014] ************************************************************* [ 0.166711] NET: Registered protocol family 16 [ 0.167472] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170470] cpuidle: using governor menu [ 0.172565] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175529] PCI: Using configuration type 1 for base access [ 0.177120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.185188] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.186029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188034] cryptd: max_cpu_qlen set to 1000 [ 0.190094] ACPI: Added _OSI(Module Device) [ 0.191015] ACPI: Added _OSI(Processor Device) [ 0.192012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.193016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.197648] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.199579] ACPI: Interpreter enabled [ 0.200076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.201014] ACPI: Using IOAPIC for interrupt routing [ 0.202094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.203440] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.212586] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.213043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.214025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.215092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.217404] acpiphp: Slot [2] registered [ 0.218129] acpiphp: Slot [5] registered [ 0.219121] acpiphp: Slot [6] registered [ 0.220116] acpiphp: Slot [3] registered [ 0.221131] acpiphp: Slot [4] registered [ 0.222117] acpiphp: Slot [7] registered [ 0.223121] acpiphp: Slot [8] registered [ 0.224125] acpiphp: Slot [9] registered [ 0.225163] acpiphp: Slot [10] registered [ 0.226107] acpiphp: Slot [11] registered [ 0.227147] acpiphp: Slot [12] registered [ 0.228113] acpiphp: Slot [13] registered [ 0.229107] acpiphp: Slot [14] registered [ 0.230103] acpiphp: Slot [15] registered [ 0.231108] acpiphp: Slot [16] registered [ 0.232108] acpiphp: Slot [17] registered [ 0.233147] acpiphp: Slot [18] registered [ 0.234120] acpiphp: Slot [19] registered [ 0.235120] acpiphp: Slot [20] registered [ 0.236113] acpiphp: Slot [21] registered [ 0.237131] acpiphp: Slot [22] registered [ 0.238104] acpiphp: Slot [23] registered [ 0.239112] acpiphp: Slot [24] registered [ 0.240139] acpiphp: Slot [25] registered [ 0.241105] acpiphp: Slot [26] registered [ 0.242131] acpiphp: Slot [27] registered [ 0.243110] acpiphp: Slot [28] registered [ 0.244134] acpiphp: Slot [29] registered [ 0.245108] acpiphp: Slot [30] registered [ 0.246160] acpiphp: Slot [31] registered [ 0.247079] PCI host bridge to bus 0000:00 [ 0.248026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.249028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.250033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.251029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.252025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.253023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.254217] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.256051] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.258180] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.262806] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.265048] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.266026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.267023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.268020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.269583] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.270777] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.271048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.272791] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.274757] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.279987] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.281016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.284966] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.287021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.290020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.297017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.303400] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.306019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.309036] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.316021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.322431] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.323472] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.324375] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.325463] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.326246] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.330029] iommu: Default domain type: Passthrough [ 0.331522] SCSI subsystem initialized [ 0.333154] ACPI: bus type USB registered [ 0.334096] usbcore: registered new interface driver usbfs [ 0.335066] usbcore: registered new interface driver hub [ 0.337085] usbcore: registered new device driver usb [ 0.338111] pps_core: LinuxPPS API ver. 1 registered [ 0.339009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.341064] PTP clock support registered [ 0.343110] EDAC MC: Ver: 3.0.0 [ 0.345094] PCI: Using ACPI for IRQ routing [ 0.346700] NetLabel: Initializing [ 0.348013] NetLabel: domain hash size = 128 [ 0.349009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.351084] NetLabel: unlabeled traffic allowed by default [ 0.353117] vgaarb: loaded [ 0.354311] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.356015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.363120] clocksource: Switched to clocksource kvm-clock [ 0.470611] VFS: Disk quotas dquot_6.6.0 [ 0.472098] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.474489] *** VALIDATE ramfs *** [ 0.475735] *** VALIDATE hugetlbfs *** [ 0.477219] pnp: PnP ACPI init [ 0.479436] pnp: PnP ACPI: found 6 devices [ 0.494987] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.497332] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.499631] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.501810] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.504311] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.506909] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.509989] NET: Registered protocol family 2 [ 0.512570] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.517207] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.520627] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.526562] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.529619] TCP: Hash tables configured (established 65536 bind 65536) [ 0.532473] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.534544] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.537106] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.539281] NET: Registered protocol family 1 [ 0.541857] RPC: Registered named UNIX socket transport module. [ 0.543985] RPC: Registered udp transport module. [ 0.545298] RPC: Registered tcp transport module. [ 0.546763] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.548911] NET: Registered protocol family 44 [ 0.550510] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.552565] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.554578] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.556667] PCI: CLS 0 bytes, default 64 [ 0.558325] Unpacking initramfs... [ 1.923610] debug: unmapping init [mem 0xffff8cd33cc64000-0xffff8cd33ffcffff] [ 1.926819] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.928845] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.931022] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.418496] Initialise system trusted keyrings [ 2.420204] Key type blacklist registered [ 2.421822] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.433203] zbud: loaded [ 2.436758] *** VALIDATE nfs *** [ 2.437660] *** VALIDATE nfs4 *** [ 2.438975] pstore: using deflate compression [ 2.441862] Platform Keyring initialized [ 2.534855] NET: Registered protocol family 38 [ 2.536791] Key type asymmetric registered [ 2.538530] Asymmetric key parser 'x509' registered [ 2.540726] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.543963] io scheduler mq-deadline registered [ 2.545512] io scheduler kyber registered [ 2.547375] io scheduler bfq registered [ 2.549547] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.553254] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.556200] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.559314] ACPI: Power Button [PWRF] [ 2.564438] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.571228] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.585322] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.613028] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.640898] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.645589] Non-volatile memory driver v1.3 [ 2.646980] Linux agpgart interface v0.103 [ 2.679448] virtio_blk virtio1: [vda] 147088 512-byte logical blocks (75.3 MB/71.8 MiB) [ 2.681618] vda: detected capacity change from 0 to 75309056 [ 2.695203] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.697249] vdb: detected capacity change from 0 to 1073741824 [ 2.708395] libphy: Fixed MDIO Bus: probed [ 2.717694] usbcore: registered new interface driver usbserial_generic [ 2.720427] usbserial: USB Serial support registered for generic [ 2.723080] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.727617] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.729593] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.732770] mousedev: PS/2 mouse device common for all mice [ 2.735259] rtc_cmos 00:05: RTC can wake from S4 [ 2.738023] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.738388] rtc_cmos 00:05: registered as rtc0 [ 2.742723] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.745743] intel_pstate: CPU model not supported [ 2.748521] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.752829] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.752889] hid: raw HID events driver (C) Jiri Kosina [ 2.757583] usbcore: registered new interface driver usbhid [ 2.759506] usbhid: USB HID core driver [ 2.761098] drop_monitor: Initializing network drop monitor service [ 2.763541] Initializing XFRM netlink socket [ 2.765563] NET: Registered protocol family 10 [ 2.768390] Segment Routing with IPv6 [ 2.769789] NET: Registered protocol family 17 [ 2.771724] mpls_gso: MPLS GSO support [ 2.777687] RAS: Correctable Errors collector initialized. [ 2.780162] AVX version of gcm_enc/dec engaged. [ 2.782077] AES CTR mode by8 optimization enabled [ 2.852092] sched_clock: Marking stable (2852071039, 0)->(3892837727, -1040766688) [ 2.856192] registered taskstats version 1 [ 2.858057] Loading compiled-in X.509 certificates [ 2.859833] zswap: loaded using pool lzo/zbud [ 2.882355] Key type big_key registered [ 2.892323] Key type encrypted registered [ 2.893904] ima: No TPM chip found, activating TPM-bypass! [ 2.896039] ima: Allocated hash algorithm: sha1 [ 2.897682] ima: No architecture policies found [ 2.899460] evm: Initialising EVM extended attributes: [ 2.901324] evm: security.selinux [ 2.902481] evm: security.ima [ 2.903423] evm: security.capability [ 2.904172] evm: HMAC attrs: 0x1 [ 2.905544] rtc_cmos 00:05: setting system clock to 2026-08-27 19:53:57 UTC (1787860437) [ 2.911146] debug: unmapping init [mem 0xffffffff8b403000-0xffffffff8b5fffff] [ 2.913814] debug: unmapping init [mem 0xffffffff8a182000-0xffffffff8a458fff] [ 2.922075] Write protecting the kernel read-only data: 28672k [ 2.924275] debug: unmapping init [mem 0xffffffff88803000-0xffffffff889fffff] [ 2.926845] debug: unmapping init [mem 0xffffffff89114000-0xffffffff891fffff] [ 2.956370] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.963836] systemd[1]: Detected virtualization kvm. [ 2.965060] systemd[1]: Detected architecture x86-64. [ 2.966118] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.990641] systemd[1]: No hostname configured. [ 2.991626] systemd[1]: Set hostname to . [ 2.992846] random: systemd: uninitialized urandom read (16 bytes read) [ 2.995399] systemd[1]: Initializing machine ID from random generator. [ 3.053498] random: ln: uninitialized urandom read (6 bytes read) [ 3.138976] random: systemd: uninitialized urandom read (16 bytes read) [ 3.141854] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.147088] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.151837] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.661556] device-mapper: uevent: version 1.0.3 [ 3.663762] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.276508] virtio_net virtio0 ens2: renamed from eth0 [ 4.310800] scsi host0: ata_piix [ 4.332720] scsi host1: ata_piix [ 4.334435] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.337871] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.003194] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.411172] random: crng init done [ 9.412931] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.652772] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.684248] printk: systemd: 25 output lines suppressed due to ratelimiting [ 10.917197] SELinux: Disabled at runtime. [ 10.972698] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.981135] systemd[1]: Detected virtualization kvm. [ 10.983135] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.451180] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.453855] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.457609] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.462501] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.466439] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.473054] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.480775] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ 11.585631] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 11.890122] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.235936] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.242441] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.312391] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.328944] EDAC sbridge: Ver: 1.1.2 [ 13.311220] Key type dns_resolver registered [ 13.586089] NFS: Registering the id_resolver key type [ 13.588036] Key type id_resolver registered [ 13.589582] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg228-client login: [ 24.832627] hrtimer: interrupt took 9819362 ns [ 75.999532] libcfs: loading out-of-tree module taints kernel. [ 76.279952] Key type ._llcrypt registered [ 76.287412] Key type .llcrypt registered [ 76.736486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 76.749325] alg: No test for adler32 (adler32-zlib) [ 78.374861] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 79.798657] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 81.717583] Key type lgssc registered [ 83.909944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 221.102288] Lustre: Mounted lustre-client - version 2.17.57_120_gb73d7ca [ 226.838391] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 246.476686] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing check_logdir /tmp/testlogs/ [ 246.755144] Lustre: lustre-OST0000-osc-ffff8cd3981c7800: disconnect after 23s idle [ 251.784163] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing yml_node [ 256.904382] Lustre: DEBUG MARKER: Client: 2.17.57.120 [ 259.105464] Lustre: DEBUG MARKER: MDS: 2.17.57.120 [ 262.363424] Lustre: DEBUG MARKER: OSS: 2.17.57.120 [ 264.010880] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Aug 27 15:58:17 EDT 2026 [ 280.908312] Lustre: Unmounted lustre-client [ 307.687886] Key type lgssc unregistered [ 308.040236] LNet: 8432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 308.045608] LNetError: 8432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 308.060374] LNet: Removed LNI 192.168.202.28@tcp [ 308.803417] Key type .llcrypt unregistered [ 308.805914] Key type ._llcrypt unregistered [ 334.881663] Key type ._llcrypt registered [ 334.888189] Key type .llcrypt registered [ 335.158263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 335.171587] alg: No test for adler32 (adler32-zlib) [ 336.226588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 336.299613] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 336.324279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 341.361261] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 341.581301] LNet: 9319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.591326] LNetError: 9319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 341.611540] LNet: Removed LNI 192.168.202.28@tcp [ 342.425497] Key type .llcrypt unregistered [ 342.431044] Key type ._llcrypt unregistered [ 350.237909] Key type ._llcrypt registered [ 350.248734] Key type .llcrypt registered [ 351.890763] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 15:59:44 (1787860784) [ 352.299323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 352.317340] alg: No test for adler32 (adler32-zlib) [ 353.227964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.323490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.353744] LNet: 9738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.382512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.435197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 353.451890] LNet: 9745:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 361.018443] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 15:59:54 (1787860794) [ 361.105550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.153825] LNet: 10317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.181822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.297085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 369.735744] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 16:00:02 (1787860802) [ 369.853231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.910584] LNet: 10903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.952537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.110248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 380.336835] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 16:00:13 (1787860813) [ 380.585819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.598075] LNet: 11497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.615533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.674723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 388.001410] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 16:00:20 (1787860820) [ 388.246188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.298685] LNet: 12080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.340722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.421931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 388.859571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 397.253476] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 16:00:30 (1787860830) [ 397.400165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.436082] LNet: 12665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.462743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.650475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 398.072729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 398.622584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 406.061872] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 16:00:39 (1787860839) [ 406.178206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.284533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.444523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 413.372744] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 16:00:46 (1787860846) [ 413.647445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.682639] LNet: 13854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.695830] LNet: 13854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 413.729709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.838882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 413.866789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 414.682242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 414.769377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 415.008908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 415.038353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 415.554610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 415.611204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 416.255989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 416.325475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 416.690308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 416.732892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 417.191910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 417.244692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 417.742744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 417.802791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 418.281630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 418.333110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 418.843647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 418.890918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 419.482634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 419.534263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 420.139509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 420.196884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 420.615732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 420.662379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 427.972672] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 16:01:00 (1787860860) [ 428.060858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.108084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.185348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 428.215333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 435.247963] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 16:01:08 (1787860868) [ 435.387581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.418701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.524861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 435.577699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 442.212744] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 16:01:15 (1787860875) [ 442.375781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.439784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.484470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 442.508661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 449.163519] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 16:01:22 (1787860882) [ 449.367147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.404223] LNet: 16307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 449.411479] LNet: 16307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 449.467908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.534226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 449.592221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 457.191265] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 16:01:30 (1787860890) [ 457.355922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.420878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.548317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 457.612192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 464.861660] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 16:01:37 (1787860897) [ 465.002385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.080537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.313350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 465.391970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 472.459224] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 16:01:45 (1787860905) [ 472.737804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.846517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.025750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.677865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 482.263676] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 16:01:55 (1787860915) [ 482.377582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.423149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.654355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 482.860239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 491.569196] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 16:02:04 (1787860924) [ 491.678489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.774297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.996971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.226593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 500.774603] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 16:02:13 (1787860933) [ 501.126416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.211173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.174868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 502.595407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 510.504970] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 16:02:23 (1787860943) [ 510.645443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.715830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.823586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.894755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 518.521959] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 16:02:31 (1787860951) [ 518.754931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.781049] LNet: 21141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 518.789435] LNet: 21141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 518.856849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.981514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 519.018561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 526.849661] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 16:02:39 (1787860959) [ 526.931410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.995633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.126479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 534.704586] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 16:02:47 (1787860967) [ 534.909849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.984878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.109337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 542.530279] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 16:02:55 (1787860975) [ 542.712393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.752969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.829838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 543.288806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 550.506480] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 16:03:03 (1787860983) [ 550.809766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.878201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.978704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 551.011524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 558.717855] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 16:03:11 (1787860991) [ 558.806324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.877806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.972933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 559.002379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 566.991360] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 16:03:20 (1787861000) [ 567.270240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.295423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.380122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 567.418557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 573.848778] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 16:03:27 (1787861007) [ 574.233930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.294864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.409810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 574.445181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 582.991143] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 16:03:35 (1787861015) [ 583.154253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.230290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.352374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 583.517732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 583.544936] LNetError: 25869:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 583.599462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 583.637659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 591.992383] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 16:03:44 (1787861024) [ 592.131703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.193741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.294371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LM9rNnKStB [ 592.389342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LM9rNnKStB [ 592.447950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LM9rNnKStB [ 600.956358] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 16:03:53 (1787861033) [ 601.097804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.158873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.263894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 601.305986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 601.352844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 609.504963] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 16:04:02 (1787861042) [ 609.749202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.825984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.923848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 609.950483] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 612.345715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 614.294191] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 616.160207] Key type lgssc registered [ 618.135863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 623.917059] LNetError: 27632:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.28@tcp99 has no available nets [ 627.469597] Key type lgssc unregistered [ 628.074656] LNetError: 28745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 628.084896] LNet: Removed LNI 192.168.202.28@tcp [ 628.921679] Key type .llcrypt unregistered [ 628.928547] Key type ._llcrypt unregistered [ 636.484709] Key type ._llcrypt registered [ 636.494736] Key type .llcrypt registered [ 637.932896] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:04:30 (1787861070) [ 638.405322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 638.417790] alg: No test for adler32 (adler32-zlib) [ 639.433171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.488826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 643.108106] LNet: Message forwarding will be enabled in 2s [ 646.176129] LNet: Message forwarding enabled [ 646.810186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 646.843586] LNet: Message forwarding will stop in 2s to 6s [ 650.429985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 652.491170] LNet: Message forwarding disabled [ 653.781314] LNet: Message forwarding will be enabled in 2s [ 657.632796] LNet: Message forwarding enabled [ 661.701265] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:04:54 (1787861094) [ 661.945254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.959144] LNet: 30146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.770880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.908449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 671.485108] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:05:04 (1787861104) [ 671.572592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.600384] LNet: 30721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.638100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.675252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 671.701502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 671.724387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 671.750926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 671.768826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 671.806225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 671.836650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 671.931331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 671.997811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 672.052584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 672.090488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 672.128393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 672.188224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 672.224733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 672.275367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 672.329189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 672.390646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 672.481529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 672.567808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 672.593516] LNetError: 30771:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 680.086524] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:05:12 (1787861112) [ 680.145667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.172727] LNet: 31337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.196382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.272043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 689.588707] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:05:22 (1787861122) [ 691.958365] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg228-server [ 694.142260] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:05:27 (1787861127) [ 694.285274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.331844] LNet: 32272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.375864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.457858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 694.490244] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 695.459707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[29-33]@tcp [ 695.487309] LNet: 32407:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 695.539981] LNetError: 32278:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.31@tcp is being used as a gateway but routing feature is not turned on [ 695.563128] LNetError: 32278:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.31@tcp has gone from up to down [ 696.549515] LNetError: 32278:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.30@tcp is being used as a gateway but routing feature is not turned on [ 696.560332] LNetError: 32278:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 697.580215] LNetError: 32278:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.33@tcp is being used as a gateway but routing feature is not turned on [ 697.595099] LNetError: 32278:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 699.635116] LNetError: 32278:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.30@tcp is being used as a gateway but routing feature is not turned on [ 699.656822] LNetError: 32278:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 703.561092] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:05:36 (1787861136) [ 703.722228] LNetError: 32278:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.33@tcp is being used as a gateway but routing feature is not turned on [ 703.737617] LNetError: 32278:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 705.459216] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg228-server [ 707.432367] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:05:40 (1787861140) [ 707.582750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.602962] LNet: 33343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.609868] LNetError: 33343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.667103] LNet: Removed LNI 192.168.202.28@tcp [ 707.691581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.739199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 707.758040] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 708.542839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[29-34/2]@tcp [ 708.575965] LNet: 33462:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 708.597633] LNet: 33462:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 708.618307] LNetError: 33349:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.31@tcp has gone from up to down [ 708.630083] LNetError: 33349:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 708.632504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[29-34/2]@tcp [ 716.197033] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:05:49 (1787861149) [ 716.288149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.306657] LNet: 34047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.319727] LNetError: 34047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 716.363367] LNet: Removed LNI 192.168.202.28@tcp [ 716.409990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.509804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 716.558671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.612817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.694511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 716.773920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.832863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.900912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 717.049199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.102320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.182444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 717.263702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.328965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.412871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 717.501915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.532421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.587602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 717.664389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.698194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.746519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 717.829315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.901298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.992205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 718.094630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.119674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.167557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 718.233356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.264929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.334400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 718.415990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.446709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.494936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 718.549322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.589442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.649398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 725.374560] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:05:58 (1787861158) [ 727.229708] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg228-server [ 729.373693] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:06:02 (1787861162) [ 731.343548] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg228-server [ 733.343125] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:06:06 (1787861166) [ 733.482699] LNet: 35448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.491686] LNet: 35448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 734.201086] Key type .llcrypt unregistered [ 734.202688] Key type ._llcrypt unregistered [ 741.199901] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 741.374657] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 741.384656] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 741.394217] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 744.353766] Key type ._llcrypt registered [ 744.355996] Key type .llcrypt registered [ 744.799726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 744.830779] alg: No test for adler32 (adler32-zlib) [ 745.747512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.855175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 745.913570] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 746.738946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 746.782735] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 749.065737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 749.093625] LNet: Removed LNI 10.1.2.3@tcp [ 749.245423] LNet: 36186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.260475] LNetError: 36186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.271914] LNet: Removed LNI 192.168.202.28@tcp [ 750.159589] Key type .llcrypt unregistered [ 750.160958] Key type ._llcrypt unregistered [ 763.261512] Key type ._llcrypt registered [ 763.263318] Key type .llcrypt registered [ 764.904643] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:06:37 (1787861197) [ 765.476060] Key type .llcrypt unregistered [ 765.480061] Key type ._llcrypt unregistered [ 772.435523] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 772.587280] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 772.589446] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 772.609591] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 775.304582] Key type ._llcrypt registered [ 775.307192] Key type .llcrypt registered [ 775.497131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 775.517458] alg: No test for adler32 (adler32-zlib) [ 776.522159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.616184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 776.658658] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 777.486615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 777.517975] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 780.411970] LNet: 37687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.418593] LNetError: 37687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 780.426872] LNet: Removed LNI 10.1.2.3@tcp [ 780.443976] LNet: Removed LNI 192.168.202.28@tcp [ 781.086717] Key type .llcrypt unregistered [ 781.089817] Key type ._llcrypt unregistered [ 793.480579] Key type ._llcrypt registered [ 793.483901] Key type .llcrypt registered [ 795.056715] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:07:07 (1787861227) [ 795.619317] Key type .llcrypt unregistered [ 795.620817] Key type ._llcrypt unregistered [ 802.284552] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 802.482631] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 802.506402] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 802.516295] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 805.248714] Key type ._llcrypt registered [ 805.250363] Key type .llcrypt registered [ 805.537628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 805.565176] alg: No test for adler32 (adler32-zlib) [ 806.530889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.592594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 806.615625] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 808.727480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 808.751459] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 811.034780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 811.065100] LNet: Removed LNI 10.1.2.3@tcp [ 811.102579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 811.129986] LNet: Removed LNI 10.1.2.31@tcp [ 811.162262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 811.212837] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 812.101656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 812.137898] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 813.097826] LNet: 39611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.103665] LNetError: 39611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 813.108627] LNet: Removed LNI 10.1.2.31@tcp [ 813.119162] LNet: Removed LNI 10.1.2.3@tcp [ 813.688609] Key type .llcrypt unregistered [ 813.691695] Key type ._llcrypt unregistered [ 826.977454] Key type ._llcrypt registered [ 826.979357] Key type .llcrypt registered [ 828.238701] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:07:41 (1787861261) [ 828.716571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 828.726354] alg: No test for adler32 (adler32-zlib) [ 829.747674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.827081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 829.858650] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 830.637381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.663265] LNet: 40568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.668694] LNetError: 40568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.692706] LNet: Removed LNI 192.168.202.28@tcp [ 830.711160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.783656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.28@tcp [ 830.804438] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 831.500645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.28@tcp [ 831.526825] LNet: Removed LNI 192.168.202.28@tcp [ 831.590892] LNet: 40704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.324453] Key type .llcrypt unregistered [ 832.326476] Key type ._llcrypt unregistered [ 845.839618] Key type ._llcrypt registered [ 845.843414] Key type .llcrypt registered [ 847.204557] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:08:00 (1787861280) [ 849.666399] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg228-server [ 851.719516] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:08:04 (1787861284) [ 852.346316] Key type .llcrypt unregistered [ 852.348607] Key type ._llcrypt unregistered [ 859.710903] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 859.956531] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 859.964087] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 859.972381] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 862.666091] Key type ._llcrypt registered [ 862.668099] Key type .llcrypt registered [ 862.832397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 862.869233] alg: No test for adler32 (adler32-zlib) [ 863.770191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.820124] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 864.428831] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 866.186194] LNet: 42548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.191430] LNetError: 42548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.196270] LNet: Removed LNI 192.168.202.28@tcp1 [ 866.210216] LNet: Removed LNI 10.1.2.3@tcp2 [ 866.869659] Key type .llcrypt unregistered [ 866.873774] Key type ._llcrypt unregistered [ 879.385352] Key type ._llcrypt registered [ 879.387291] Key type .llcrypt registered [ 880.794475] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:08:33 (1787861313) [ 881.416758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 881.436514] alg: No test for adler32 (adler32-zlib) [ 882.354697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.409482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 882.435260] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 883.465248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.485379] LNet: 43522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.490906] LNetError: 43522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.505890] LNet: Removed LNI 192.168.202.28@tcp [ 883.535949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.623884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 883.654752] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 891.259412] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:08:44 (1787861324) [ 891.393801] LNet: 44108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.398621] LNetError: 44108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.418875] LNet: Removed LNI 192.168.202.28@tcp [ 892.188623] Key type .llcrypt unregistered [ 892.196398] Key type ._llcrypt unregistered [ 899.988704] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 900.109657] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 900.119067] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 900.127388] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 903.027397] Key type ._llcrypt registered [ 903.030916] Key type .llcrypt registered [ 903.374644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 903.392684] alg: No test for adler32 (adler32-zlib) [ 904.356765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.424275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 904.468868] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 905.539947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 905.562348] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 908.097830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.132965] LNet: 44863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.150327] LNetError: 44863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 908.153249] LNet: Removed LNI 10.1.2.3@tcp [ 908.189263] LNet: Removed LNI 192.168.202.28@tcp [ 908.234105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.322684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 908.363328] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 908.366934] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 908.957634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.987691] LNet: 44890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.000670] LNetError: 44890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 909.014088] LNetError: 44890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 909.020110] LNet: Removed LNI 10.1.2.3@tcp [ 909.034456] LNet: Removed LNI 192.168.202.28@tcp [ 909.064285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.175305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 909.203331] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 909.209391] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 909.746509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 909.779976] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 910.787612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 910.806593] LNet: 45052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.817555] LNetError: 45052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 910.824667] LNetError: 45052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 910.833891] LNet: Removed LNI 10.1.2.3@tcp [ 910.836568] LNet: Removed LNI 192.168.202.28@tcp [ 910.850210] LNet: Removed LNI 192.168.202.28@tcp2 [ 910.881748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.949190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 910.977364] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 910.983588] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 910.990372] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 918.950104] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:09:11 (1787861351) [ 919.185915] LNet: 45649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.199161] LNetError: 45649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 919.206439] LNetError: 45649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 919.224119] LNet: Removed LNI 10.1.2.3@tcp [ 919.229520] LNet: Removed LNI 192.168.202.28@tcp [ 919.253137] LNet: Removed LNI 192.168.202.28@tcp2 [ 919.922679] Key type .llcrypt unregistered [ 919.923724] Key type ._llcrypt unregistered [ 927.425721] Key type ._llcrypt registered [ 927.429381] Key type .llcrypt registered [ 927.668345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 927.688425] alg: No test for adler32 (adler32-zlib) [ 928.660284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.759477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 928.789130] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 929.492746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.513652] LNet: 46047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.518655] LNetError: 46047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.528719] LNet: Removed LNI 192.168.202.28@tcp [ 929.541722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.588406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 929.613564] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 930.380347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.428395] LNet: 46077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.432675] LNetError: 46077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.455676] LNet: Removed LNI 192.168.202.28@tcp [ 930.483965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.568811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 930.584834] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 931.262059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.307315] LNet: Removed LNI 192.168.202.28@tcp [ 931.349042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.460299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 931.499908] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 932.277117] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 932.463784] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 932.475055] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 932.496612] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 934.940242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.957477] LNet: 46180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.963061] LNet: 46180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 934.969742] LNetError: 46180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.976847] LNetError: 46180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 934.995061] LNet: Removed LNI 192.168.202.28@tcp [ 935.014754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.103663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 935.135445] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 942.393813] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:09:35 (1787861375) [ 942.733700] LNet: 46779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.740921] LNetError: 46779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.769865] LNet: Removed LNI 192.168.202.28@tcp [ 943.479198] Key type .llcrypt unregistered [ 943.481862] Key type ._llcrypt unregistered [ 951.131621] Key type ._llcrypt registered [ 951.136960] Key type .llcrypt registered [ 951.463662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 951.503352] alg: No test for adler32 (adler32-zlib) [ 952.483883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.536839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 952.583385] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 953.889628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.901657] LNet: 47182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.916086] LNetError: 47182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.936964] LNet: Removed LNI 192.168.202.28@tcp [ 953.964889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.049192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 954.083572] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 961.108918] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:09:54 (1787861394) [ 961.335107] LNet: 47768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.339862] LNetError: 47768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.376233] LNet: Removed LNI 192.168.202.28@tcp [ 962.413548] Key type .llcrypt unregistered [ 962.420834] Key type ._llcrypt unregistered [ 970.024477] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 970.183962] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 970.213692] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 970.216363] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 972.885684] Key type ._llcrypt registered [ 972.887964] Key type .llcrypt registered [ 973.040679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 973.059755] alg: No test for adler32 (adler32-zlib) [ 973.979850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.049833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 974.106470] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 975.164871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.183349] LNet: 48229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.188923] LNetError: 48229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.215575] LNet: Removed LNI 192.168.202.28@tcp [ 975.233675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.334852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 975.367799] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 982.357799] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:10:15 (1787861415) [ 982.620300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.642666] LNet: 48827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.648685] LNetError: 48827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.660330] LNet: Removed LNI 192.168.202.28@tcp [ 982.689838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.804326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 990.133565] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:10:23 (1787861423) [ 990.335425] LNet: 49409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.095712] Key type .llcrypt unregistered [ 991.098748] Key type ._llcrypt unregistered [ 999.011293] Key type ._llcrypt registered [ 999.016947] Key type .llcrypt registered [ 999.208843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 999.232945] alg: No test for adler32 (adler32-zlib) [ 1000.147827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.198384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1000.219837] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1000.945548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1000.954823] LNet: 49811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.958357] LNetError: 49811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1000.970374] LNet: Removed LNI 192.168.202.28@tcp [ 1000.985188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1001.040803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1001.059971] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1007.861357] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:10:41 (1787861441) [ 1007.976557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1008.009865] LNet: 50397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.019539] LNetError: 50397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.043759] LNet: Removed LNI 192.168.202.28@tcp [ 1008.062196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.124728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1014.877342] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:10:47 (1787861447) [ 1016.760874] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1019.145905] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:10:51 (1787861451) [ 1020.676716] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1022.412553] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:10:55 (1787861455) [ 1023.957962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1025.582196] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:10:58 (1787861458) [ 1027.090574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1028.978800] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:11:01 (1787861461) [ 1030.473476] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1032.312706] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 16:11:05 (1787861465) [ 1032.663140] LNet: 52459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.569758] Key type .llcrypt unregistered [ 1033.572377] Key type ._llcrypt unregistered [ 1039.753796] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1039.937919] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1039.941069] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1039.944101] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1042.302444] Key type ._llcrypt registered [ 1042.304099] Key type .llcrypt registered [ 1042.440788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1042.453680] alg: No test for adler32 (adler32-zlib) [ 1043.327729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.382919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.408816] LNet: Added LNI 192.168.202.28@tcp [8/512/0/180] [ 1043.412239] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1043.414793] LNet: 52778:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1043.480040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.514703] LNet: 52797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.519361] LNetError: 52797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1043.523851] LNet: Removed LNI 10.1.2.3@tcp [ 1043.527104] LNetError: 52797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.554240] LNet: Removed LNI 192.168.202.28@tcp [ 1043.576924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.684542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.722290] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1043.775715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.797669] LNet: 52821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.801666] LNetError: 52821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.817935] LNet: Removed LNI 192.168.202.28@tcp [ 1043.842697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.906085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.919649] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1043.922377] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1043.923857] LNet: 52829:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1043.973409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.988397] LNet: 52847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.000925] LNetError: 52847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1044.015265] LNet: Removed LNI 10.1.2.3@tcp [ 1044.018499] LNetError: 52847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.034358] LNet: Removed LNI 192.168.202.28@tcp [ 1044.064673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1044.124956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1044.151269] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1044.155717] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1044.157847] LNet: 52855:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1044.330657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1044.349426] LNet: 52881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.355308] LNetError: 52881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1044.357662] LNet: Removed LNI 10.1.2.3@tcp [ 1044.361096] LNetError: 52881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.372868] LNet: Removed LNI 192.168.202.28@tcp [ 1044.385539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1044.429498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1044.445062] LNet: Added LNI 192.168.202.28@tcp [8/256/0/182] [ 1044.449581] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1044.548639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1044.566494] LNet: 52909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.571401] LNetError: 52909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1044.580292] LNet: Removed LNI 10.1.2.3@tcp [ 1044.586310] LNetError: 52909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.597846] LNet: Removed LNI 192.168.202.28@tcp [ 1044.618712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1044.684917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1044.709592] LNet: Added LNI 192.168.202.28@tcp [8/256/0/182] [ 1044.712783] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1044.714607] LNet: 52917:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1044.810657] LNet: Removed LNI 10.1.2.3@tcp [ 1044.827390] LNet: Removed LNI 192.168.202.28@tcp [ 1045.580921] Key type .llcrypt unregistered [ 1045.584915] Key type ._llcrypt unregistered [ 1056.275585] Key type ._llcrypt registered [ 1056.277644] Key type .llcrypt registered [ 1058.287096] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 16:11:30 (1787861490) [ 1060.625564] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1063.377740] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:11:35 (1787861495) [ 1064.260863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1064.296284] alg: No test for adler32 (adler32-zlib) [ 1065.396237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1065.488662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1065.510980] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1066.896802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.28 [ 1066.940866] LNet: 54188:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1066.948981] LNet: Removed LNI 192.168.202.28@tcp [ 1075.528813] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:11:48 (1787861508) [ 1075.804247] LNet: 54754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.819646] LNetError: 54754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1075.852281] LNet: Removed LNI 192.168.202.28@tcp [ 1077.042928] Key type .llcrypt unregistered [ 1077.050667] Key type ._llcrypt unregistered [ 1083.396052] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1083.817067] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1086.738480] Key type ._llcrypt registered [ 1086.744512] Key type .llcrypt registered [ 1086.922756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1086.934845] alg: No test for adler32 (adler32-zlib) [ 1088.329639] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1096.437842] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:12:09 (1787861529) [ 1096.538836] LNet: 55669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.555438] LNetError: 55669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1096.614968] LNet: Removed LNI 10.1.2.3@tcp [ 1097.630914] Key type .llcrypt unregistered [ 1097.641596] Key type ._llcrypt unregistered [ 1106.198062] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1106.556673] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1106.568195] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1106.583554] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1108.329376] Key type ._llcrypt registered [ 1108.332742] Key type .llcrypt registered [ 1108.472982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1108.484025] alg: No test for adler32 (adler32-zlib) [ 1109.488128] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1117.076031] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:12:30 (1787861550) [ 1118.775170] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1120.748546] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:12:33 (1787861553) [ 1121.069099] LNet: 56858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1121.086494] LNetError: 56858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1121.119510] LNet: Removed LNI 10.1.2.3@tcp [ 1121.953567] Key type .llcrypt unregistered [ 1121.955250] Key type ._llcrypt unregistered [ 1130.026830] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1130.301981] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1130.310236] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1130.326036] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1133.242817] Key type ._llcrypt registered [ 1133.244775] Key type .llcrypt registered [ 1133.609496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1133.636602] alg: No test for adler32 (adler32-zlib) [ 1134.547711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1134.599119] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1144.457654] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:12:57 (1787861577) [ 1145.214946] LNet: 58094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.240871] LNetError: 58094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1145.277140] LNet: Removed LNI 10.1.2.3@tcp [ 1146.282772] Key type .llcrypt unregistered [ 1146.284338] Key type ._llcrypt unregistered [ 1154.928569] Key type ._llcrypt registered [ 1154.933505] Key type .llcrypt registered [ 1155.278996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1155.331115] alg: No test for adler32 (adler32-zlib) [ 1156.355950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1156.414750] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1164.888547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1165.819959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_interrupt -r 1 [ 1165.865673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_interrupt -r 1 [ 1165.900994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1165.918565] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1166.187298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.236635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1166.303400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1166.345309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1166.389970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1166.421529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1166.448327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1166.481369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.507771] LNet: 58536:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1166.546371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.596957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.629781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.686738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1166.694385] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1166.711410] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1166.760964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.799291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.855449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1167.428157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_dropped -r 1 [ 1167.476656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_dropped -r 1 [ 1167.528702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1167.861995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.901646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1167.989529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1168.040914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.084762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.130930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.214788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.265800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.312971] LNet: 58646:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.328279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.372629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.410714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.444411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.484677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.541352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.618854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1169.290460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_aborted -r 1 [ 1169.336937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_aborted -r 1 [ 1169.385802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1169.426072] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1169.433848] LNet: Skipped 1 previous similar message [ 1169.759710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.805947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1169.873855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.919950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.956992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.980724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.026062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.076925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.125399] LNet: 58755:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1170.166500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.225695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.277752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.317406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.360409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.424946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.483238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1171.046850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_no_route -r 1 [ 1171.088986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_no_route -r 1 [ 1171.125854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1171.351589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.398318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.465639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.510414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.543979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.580895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.613341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.643826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.697540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.776127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.820511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.895231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.933094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.978572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.015485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.672395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_timeout -r 1 [ 1172.715606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_timeout -r 1 [ 1172.766755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1172.791291] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1172.795507] LNet: Skipped 1 previous similar message [ 1173.061208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.092774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.140483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.197783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.231149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.278128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.309761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.341584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.371256] LNet: 58974:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.385169] LNet: 58974:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1173.420371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.470674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.537743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.582973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.631243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.670506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.708407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1174.099325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_error -r 1 [ 1174.137623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_error -r 1 [ 1174.184270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1174.426449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.458266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.483461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.517402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.560105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.592673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.632587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.673969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.712851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.745462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.779491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.809442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.846874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.874819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.903305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1175.138572] LNet: 59128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.145466] LNetError: 59128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.166593] LNet: Removed LNI 192.168.202.28@tcp [ 1175.634614] Key type .llcrypt unregistered [ 1175.637188] Key type ._llcrypt unregistered [ 1194.131479] Key type ._llcrypt registered [ 1194.133674] Key type .llcrypt registered [ 1195.934750] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 16:13:48 (1787861628) [ 1196.783676] Key type .llcrypt unregistered [ 1196.788966] Key type ._llcrypt unregistered [ 1204.458674] Key type ._llcrypt registered [ 1204.461039] Key type .llcrypt registered [ 1204.747183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1204.795413] alg: No test for adler32 (adler32-zlib) [ 1205.717041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1205.781595] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1213.692092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1223.992878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1224.068465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1224.090178] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1225.492251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_interrupt -r 1 [ 1225.531241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_interrupt -r 1 [ 1225.590889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_interrupt -r 1 [ 1225.651446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_interrupt -r 1 [ 1225.712949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_interrupt -r 1 [ 1225.755550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_interrupt -r 1 [ 1225.821310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_interrupt -r 1 [ 1225.881907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_interrupt -r 1 [ 1225.919937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1225.947287] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1225.963112] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1226.289183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1226.313771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1226.344412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.390289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.481789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1226.505950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1226.536427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1226.570780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.589813] LNet: 60603:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.610235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.677858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.800437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.861193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.912989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.945909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.992589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1227.600880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_dropped -r 1 [ 1227.653268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_dropped -r 1 [ 1227.695791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_dropped -r 1 [ 1227.746987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_dropped -r 1 [ 1227.786244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_dropped -r 1 [ 1227.860846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_dropped -r 1 [ 1227.955861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_dropped -r 1 [ 1228.050481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_dropped -r 1 [ 1228.114615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1228.139837] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1228.145355] LNet: Skipped 2 previous similar messages [ 1228.370499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1228.394249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1228.430353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.463427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.506994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.535653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.565361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.606751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.625249] LNet: 60725:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1228.660086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1228.695211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1228.747938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.784188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.822779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.879955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1228.938588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1229.710575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_aborted -r 1 [ 1229.789714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_aborted -r 1 [ 1229.843620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_aborted -r 1 [ 1229.877392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_aborted -r 1 [ 1229.909116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_aborted -r 1 [ 1229.944927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_aborted -r 1 [ 1229.983610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_aborted -r 1 [ 1230.030677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_aborted -r 1 [ 1230.063060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1230.084153] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1230.090849] LNet: Skipped 2 previous similar messages [ 1230.373388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1230.417818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1230.473639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1230.543266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1230.587725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1230.629688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1230.657345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1230.689592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.704750] LNet: 60846:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1230.723710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1230.752283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.790034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.822794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.880692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.917582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1230.950942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1231.610253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_no_route -r 1 [ 1231.672535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_no_route -r 1 [ 1231.749704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_no_route -r 1 [ 1231.814335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_no_route -r 1 [ 1231.858176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_no_route -r 1 [ 1231.928026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_no_route -r 1 [ 1232.001298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_no_route -r 1 [ 1232.055274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_no_route -r 1 [ 1232.089781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1232.103614] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1232.110942] LNet: Skipped 2 previous similar messages [ 1232.345354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1232.407115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1232.455887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1232.517590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1232.578776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1232.619654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1232.663942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1232.712419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1232.736666] LNet: 60967:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1232.779642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1232.843784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1232.881956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1232.933776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1233.002533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.026187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1233.090273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1233.849781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_timeout -r 1 [ 1233.886750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_timeout -r 1 [ 1233.930093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_timeout -r 1 [ 1233.972358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_timeout -r 1 [ 1234.022298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_timeout -r 1 [ 1234.073972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_timeout -r 1 [ 1234.116042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_timeout -r 1 [ 1234.173586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_timeout -r 1 [ 1234.241578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1234.614717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1234.658140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1234.692307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1234.744517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1234.781651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1234.836324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1234.866472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1234.920471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1234.956171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1235.005900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1235.068055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1235.121504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1235.164917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1235.207717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1235.241159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1235.834178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_error -r 1 [ 1235.875474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_error -r 1 [ 1235.908881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_error -r 1 [ 1235.957877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_error -r 1 [ 1236.014736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_error -r 1 [ 1236.044792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_error -r 1 [ 1236.076286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_error -r 1 [ 1236.117097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_error -r 1 [ 1236.196734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1236.224754] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1236.231503] LNet: Skipped 5 previous similar messages [ 1236.492081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.543803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1236.607454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.654897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.708395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.758331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.811205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.870798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.893794] LNet: 61210:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.898246] LNet: 61210:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1236.917735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.977703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1237.027535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1237.066197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1237.103761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1237.138598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1237.176397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1243.336896] LNet: 61290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.342893] LNetError: 61290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1243.352481] LNet: Removed LNI 192.168.202.28@tcp [ 1243.369241] LNet: Removed LNI 192.168.202.28@tcp1 [ 1243.752312] Key type .llcrypt unregistered [ 1243.757307] Key type ._llcrypt unregistered [ 1263.712864] Key type ._llcrypt registered [ 1263.714579] Key type .llcrypt registered [ 1265.679950] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 16:14:58 (1787861698) [ 1266.871329] Key type .llcrypt unregistered [ 1266.873876] Key type ._llcrypt unregistered [ 1275.516610] Key type ._llcrypt registered [ 1275.524362] Key type .llcrypt registered [ 1275.695319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1275.703761] alg: No test for adler32 (adler32-zlib) [ 1276.759695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1276.813479] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1285.602303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1286.318440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_dropped -r 1 [ 1286.358340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_dropped -r 1 [ 1286.394831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1286.413383] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1286.697807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1286.753967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1286.817991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1286.869426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1286.930832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1286.963967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1287.005613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1287.125526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1287.155631] LNet: 62546:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1287.179954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1287.207146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1287.250387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1287.277430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1287.303455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1287.349082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1287.416134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1287.939841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_error -r 1 [ 1287.972224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_error -r 1 [ 1288.030607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1288.052224] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1288.286661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1288.308603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1288.334506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1288.360542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1288.383287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1288.405952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1288.429480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1288.454458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1288.479288] LNet: 62655:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1288.508472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1288.541984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1288.564980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1288.600097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1288.654991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.722059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1288.761772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1289.457545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_timeout -r 1 [ 1289.520984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_timeout -r 1 [ 1289.578796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1289.617048] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1290.011731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1290.042674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1290.065823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1290.097262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1290.125613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1290.155731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1290.192635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1290.229259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1290.250911] LNet: 62764:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1290.297130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1290.353835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1290.394210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1290.432829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1290.461371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.488557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1290.512276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1290.777283] LNet: 62809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.782309] LNetError: 62809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1290.806964] LNet: Removed LNI 192.168.202.28@tcp [ 1291.197393] Key type .llcrypt unregistered [ 1291.199298] Key type ._llcrypt unregistered [ 1310.595224] Key type ._llcrypt registered [ 1310.597594] Key type .llcrypt registered [ 1312.128479] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 16:15:44 (1787861744) [ 1313.115576] Key type .llcrypt unregistered [ 1313.120210] Key type ._llcrypt unregistered [ 1320.067985] Key type ._llcrypt registered [ 1320.069540] Key type .llcrypt registered [ 1320.301328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1320.324322] alg: No test for adler32 (adler32-zlib) [ 1321.261691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1321.295335] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1330.099391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1339.627325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1339.669438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1339.701398] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1340.734643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_dropped -r 1 [ 1340.767921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e remote_dropped -r 1 [ 1340.807714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_dropped -r 1 [ 1340.839565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e remote_dropped -r 1 [ 1340.868807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e remote_dropped -r 1 [ 1340.900684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e remote_dropped -r 1 [ 1340.929726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e remote_dropped -r 1 [ 1340.962496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e remote_dropped -r 1 [ 1340.995214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1341.014419] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1341.019763] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp [ 1341.235970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1341.280754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1341.334087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1341.360972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1341.424324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1341.484562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1341.529873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1341.571157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1341.594670] LNet: 64282:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1341.617778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1341.663308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1341.707259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.746598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1341.791560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.833395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1341.880249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1341.911191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1342.491797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_error -r 1 [ 1342.532663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e remote_error -r 1 [ 1342.565420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_error -r 1 [ 1342.592866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e remote_error -r 1 [ 1342.623168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e remote_error -r 1 [ 1342.653944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e remote_error -r 1 [ 1342.696985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e remote_error -r 1 [ 1342.741701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e remote_error -r 1 [ 1342.796315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1342.817480] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1342.821919] LNet: Skipped 2 previous similar messages [ 1343.084157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1343.110958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1343.148327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1343.172914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1343.213390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1343.247251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1343.291107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1343.332677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1343.367282] LNet: 64406:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1343.386954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1343.405310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1343.442083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1343.464908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1343.484202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1343.505440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1343.542438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1343.577795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1344.045668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_timeout -r 1 [ 1344.088282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e remote_timeout -r 1 [ 1344.124250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_timeout -r 1 [ 1344.166729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e remote_timeout -r 1 [ 1344.208917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e remote_timeout -r 1 [ 1344.271384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e remote_timeout -r 1 [ 1344.304956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e remote_timeout -r 1 [ 1344.360807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e remote_timeout -r 1 [ 1344.428223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1344.464430] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1344.468613] LNet: Skipped 1 previous similar message [ 1344.773420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1344.815322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1344.891359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1344.984060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1345.043756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1345.073771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1345.124143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1345.164081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1345.218743] LNet: 64529:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1345.253552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1345.317391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1345.375159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1345.416994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1345.455067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1345.497316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1345.534564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1345.574718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.946387] LNet: 64611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.950921] LNetError: 64611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.958072] LNet: Removed LNI 192.168.202.28@tcp [ 1349.966207] LNet: Removed LNI 192.168.202.28@tcp1 [ 1350.391617] Key type .llcrypt unregistered [ 1350.393223] Key type ._llcrypt unregistered [ 1369.282842] Key type ._llcrypt registered [ 1369.285131] Key type .llcrypt registered [ 1370.668477] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 16:16:43 (1787861803) [ 1372.095701] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1374.121473] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 16:16:47 (1787861807) [ 1375.567337] Key type .llcrypt unregistered [ 1375.573396] Key type ._llcrypt unregistered [ 1383.572598] Key type ._llcrypt registered [ 1383.577362] Key type .llcrypt registered [ 1383.977731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1384.032391] alg: No test for adler32 (adler32-zlib) [ 1384.990628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1385.020794] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1392.929092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1394.045278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e network_timeout -r 1 [ 1394.079566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e network_timeout -r 1 [ 1394.136147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1394.200741] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1394.274184] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1394.640712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1394.699553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1394.775729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1394.849867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1394.886660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1394.943942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1394.992246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1395.043980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1395.074542] LNet: 66161:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1395.109361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1395.149237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1395.190833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1395.263648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1395.302056] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1395.351945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1395.422678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1395.961335] LNet: 66204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1395.970742] LNetError: 66204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1395.997318] LNet: Removed LNI 192.168.202.28@tcp [ 1396.649293] Key type .llcrypt unregistered [ 1396.652291] Key type ._llcrypt unregistered [ 1420.180900] Key type ._llcrypt registered [ 1420.182426] Key type .llcrypt registered [ 1420.586512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1420.625551] alg: No test for adler32 (adler32-zlib) [ 1421.645214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1421.690968] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1430.381504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1440.947767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1441.039031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1441.086554] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1442.814036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e network_timeout -r 1 [ 1442.860568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e network_timeout -r 1 [ 1442.931856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e network_timeout -r 1 [ 1442.996670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e network_timeout -r 1 [ 1443.043734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e network_timeout -r 1 [ 1443.077362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e network_timeout -r 1 [ 1443.110960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e network_timeout -r 1 [ 1443.172346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e network_timeout -r 1 [ 1443.203746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1443.222677] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1443.232086] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1443.239257] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp [ 1443.536163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1443.579226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1443.618479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1443.653935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1443.685412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1443.719389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1443.759130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1443.820299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1443.852623] LNet: 67090:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1443.901902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1443.963910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1444.037908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1444.069135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1444.098325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1444.131256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1449.623244] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1450.075426] LNet: 67169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.079988] LNetError: 67169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.104263] LNet: Removed LNI 192.168.202.28@tcp [ 1451.106449] LNetError: 67169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.134153] LNet: Removed LNI 192.168.202.28@tcp1 [ 1451.697585] Key type .llcrypt unregistered [ 1451.700066] Key type ._llcrypt unregistered [ 1471.116470] Key type ._llcrypt registered [ 1471.118556] Key type .llcrypt registered [ 1472.286461] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 16:18:25 (1787861905) [ 1472.801992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1472.820333] alg: No test for adler32 (adler32-zlib) [ 1473.804709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1473.883970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1473.940799] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1474.712994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1474.739658] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1475.470940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1475.598896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1475.658814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1475.720878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1475.770604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1475.807824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1475.855487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.28@tcp [ 1475.881847] LNetError: 68273:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.28@tcp to recovery [ 1476.897779] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1476.908763] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1478.944671] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1479.464368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1482.016905] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1483.064800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1483.111183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1483.175228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1483.187890] LNet: 68380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1483.191601] LNetError: 68380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1483.194796] LNet: Removed LNI 192.168.202.28@tcp [ 1483.200881] LNet: Removed LNI 192.168.202.28@tcp1 [ 1483.213510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.254867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1483.285634] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1483.898352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1483.910685] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1484.856631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1484.913690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1485.011560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1485.059769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1485.110586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1485.147783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1485.212874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1485.269665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.28@tcp [ 1485.316973] LNetError: 68653:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.28@tcp to recovery [ 1486.304419] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1488.947658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1492.448356] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1492.462235] LNet: Skipped 1 previous similar message [ 1496.911252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1496.939898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1496.976628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1505.525618] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 16:18:58 (1787861938) [ 1505.636193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1505.667149] LNet: 69367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1505.676033] LNetError: 69367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1505.681620] LNetError: 69367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1505.693627] LNet: Removed LNI 192.168.202.28@tcp [ 1505.725522] LNet: Removed LNI 192.168.202.28@tcp1 [ 1505.761363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1505.826656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1505.878728] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1506.791542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1506.814666] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1507.623706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1507.707418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1507.758663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1507.796739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1507.851458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1507.915784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1507.945151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.28@tcp --health 0 [ 1508.068338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1508.896423] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.28@tcp [ 1508.900617] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1508.914138] LNet: Skipped 1 previous similar message [ 1522.933788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1523.150122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1523.174618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1523.195435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.28@tcp --health 500 [ 1523.334341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1530.217422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1530.305597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1530.344125] LNet: 69854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.352452] LNetError: 69854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.365035] LNetError: 69854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1530.382228] LNet: Removed LNI 192.168.202.28@tcp [ 1530.413538] LNet: Removed LNI 192.168.202.28@tcp1 [ 1530.509494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.635612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1530.736319] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1531.793767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1531.825395] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1533.310820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1533.410708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1533.529176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1533.581956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1533.666408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1533.759553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1533.821528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1533.883286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.28@tcp --health 0 [ 1534.694070] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1534.706106] LNet: Skipped 3 previous similar messages [ 1535.377478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1546.370781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1546.421340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1546.471674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1554.103319] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 16:19:47 (1787861987) [ 1554.466972] LNet: 70901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.472826] LNetError: 70901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.479082] LNetError: 70901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1554.489810] LNet: Removed LNI 192.168.202.28@tcp [ 1554.506768] LNet: Removed LNI 192.168.202.28@tcp1 [ 1555.134916] Key type .llcrypt unregistered [ 1555.139198] Key type ._llcrypt unregistered [ 1563.569233] Key type ._llcrypt registered [ 1563.571055] Key type .llcrypt registered [ 1563.792704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1563.832688] alg: No test for adler32 (adler32-zlib) [ 1564.758501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1564.803898] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1574.666789] Lustre: DEBUG MARKER: Initial discovery [ 1575.002162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.128@tcp [ 1578.468538] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1578.811746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1578.891310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.128@tcp [ 1578.940963] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1578.973147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1579.037547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.128@tcp -r 1 -m GET -l 3 [ 1579.081963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.128@tcp -r 1 -m GET -e local_error [ 1579.149967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.128@tcp -r 1 -m PUT -l 6 [ 1579.168330] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1580.908462] Lustre: DEBUG MARKER: Force oleg228-client.virtnet to discover 192.168.202.128@tcp (in background) [ 1581.104768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.128@tcp [ 1582.129702] Lustre: DEBUG MARKER: Force oleg228-server to discover 192.168.202.28@tcp [ 1584.161465] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1584.991665] Lustre: DEBUG MARKER: Wait for 71671 [ 1586.772965] Lustre: DEBUG MARKER: Finished wait on 71671 [ 1586.878977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1586.914541] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1586.917276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1587.208273] LNet: 72123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.212870] LNetError: 72123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.264847] LNet: Removed LNI 192.168.202.28@tcp [ 1588.850829] Key type .llcrypt unregistered [ 1588.859654] Key type ._llcrypt unregistered [ 1608.416748] Key type ._llcrypt registered [ 1608.418782] Key type .llcrypt registered [ 1609.692361] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 16:20:42 (1787862042) [ 1610.292281] Key type .llcrypt unregistered [ 1610.293962] Key type ._llcrypt unregistered [ 1617.426692] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1617.660778] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1617.669461] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1617.677613] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1620.208398] Key type ._llcrypt registered [ 1620.219053] Key type .llcrypt registered [ 1620.428804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1620.442218] alg: No test for adler32 (adler32-zlib) [ 1621.404687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.503485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1621.551146] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1622.503974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1622.532504] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1631.771770] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 16:21:04 (1787862064) [ 1631.926602] LNet: 74198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.935844] LNetError: 74198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1631.945501] LNet: Removed LNI 10.1.2.3@tcp [ 1631.958268] LNet: Removed LNI 192.168.202.28@tcp [ 1632.610490] Key type .llcrypt unregistered [ 1632.612640] Key type ._llcrypt unregistered [ 1641.109648] Key type ._llcrypt registered [ 1641.111194] Key type .llcrypt registered [ 1641.446513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1641.479761] alg: No test for adler32 (adler32-zlib) [ 1642.347378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.381925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1642.397841] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1643.070368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1643.083235] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1643.864214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.28@tcp [ 1644.897612] LNetError: Unexpected error -22 connecting to 192.168.202.28@tcp at host 192.168.202.28:988 [ 1652.248863] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 16:21:24 (1787862084) [ 1652.413679] LNet: 75332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.433032] LNetError: 75332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1652.446314] LNet: Removed LNI 10.1.2.3@tcp [ 1652.474782] LNet: Removed LNI 192.168.202.28@tcp [ 1653.353836] Key type .llcrypt unregistered [ 1653.357648] Key type ._llcrypt unregistered [ 1662.592934] Key type ._llcrypt registered [ 1662.599556] Key type .llcrypt registered [ 1662.840904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1662.863311] alg: No test for adler32 (adler32-zlib) [ 1663.889260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.982566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1664.044662] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1665.613169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1665.682638] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 1667.163963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.28@tcp1 --nid 192.168.202.28@tcp2 [ 1673.946931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp1 192.168.202.28@tcp2 [ 1673.966720] LNetError: 76181:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.28@tcp1 to 192.168.202.28@tcp2 - Route table may be misconfigured [ 1673.986477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp2 192.168.202.28@tcp1 [ 1682.979584] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 16:21:55 (1787862115) [ 1683.113897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.155756] LNet: 76750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.183993] LNetError: 76750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.257128] LNet: Removed LNI 192.168.202.28@tcp1 [ 1684.259108] LNetError: 76750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.293429] LNet: Removed LNI 192.168.202.28@tcp2 [ 1684.374928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.479896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1684.554705] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1685.302222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1685.315659] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1685.849169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1685.866705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.28@tcp [ 1685.882405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp1 192.168.202.28@tcp1 [ 1685.899053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.28@tcp -d 192.168.202.28@tcp -e network_timeout [ 1685.922861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -e network_timeout [ 1685.942277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -e network_timeout [ 1685.961738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -e network_timeout [ 1685.981343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1685.991505] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1686.002810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1686.058454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1686.113758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1686.503679] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1693.155754] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 16:22:05 (1787862125) [ 1693.247637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1693.277151] LNet: 77609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1693.287604] LNetError: 77609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.304217] LNet: Removed LNI 192.168.202.28@tcp [ 1695.336089] LNet: Removed LNI 192.168.202.28@tcp1 [ 1695.429180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.651183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1696.087463] LNet: 77660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.558755] Key type .llcrypt unregistered [ 1696.565464] Key type ._llcrypt unregistered [ 1711.304441] Key type ._llcrypt registered [ 1711.306443] Key type .llcrypt registered [ 1713.051717] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 16:22:25 (1787862145) [ 1713.461407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1713.476808] alg: No test for adler32 (adler32-zlib) [ 1714.463877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.157967] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1718.376210] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 16:22:31 (1787862151) [ 1718.548908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1718.573615] LNet: 78753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.602649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.672610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1718.692821] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1719.558940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1719.590372] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1720.789240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1720.859934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp1 [ 1720.937190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp1 [ 1728.619659] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 16:22:41 (1787862161) [ 1730.821685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg228-server [ 1732.523519] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 16:22:45 (1787862165) [ 1734.479992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg228-server [ 1736.484610] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 16:22:49 (1787862169) [ 1738.965996] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg228-server [ 1740.533351] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 16:22:53 (1787862173) [ 1742.497684] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg228-server [ 1744.726501] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 16:22:57 (1787862177) [ 1746.509820] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg228-server [ 1748.324177] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 16:23:01 (1787862181) [ 1750.612616] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg228-server [ 1752.657714] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 16:23:05 (1787862185) [ 1754.608710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg228-server [ 1756.583946] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 16:23:09 (1787862189) [ 1758.457608] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg228-server [ 1759.936216] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 16:23:13 (1787862193) [ 1761.520279] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg228-server [ 1763.154220] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 16:23:16 (1787862196) [ 1763.481288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.501637] LNet: 82636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.511834] LNetError: 82636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.512228] LNet: Removed LNI 192.168.202.28@tcp [ 1764.518265] LNetError: 82636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.546270] LNet: Removed LNI 192.168.202.28@tcp1 [ 1764.592174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.695583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1764.732227] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1765.424213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1765.516189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1765.717701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.765537] LNet: 82783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.772192] LNetError: 82783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1766.828693] LNet: Removed LNI 192.168.202.28@tcp [ 1766.862672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.920509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1766.967026] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1767.879695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1768.052262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.065723] LNet: 82929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1768.071880] LNetError: 82929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1768.086208] LNet: Removed LNI 192.168.202.28@tcp [ 1768.113430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.197725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1768.229520] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1769.060144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1769.130891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1769.310839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.349238] LNet: Removed LNI 192.168.202.28@tcp [ 1770.438376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.544913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1770.590938] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1771.444726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1771.744831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.839573] LNet: Removed LNI 192.168.202.28@tcp [ 1772.921463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.072680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1773.130058] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1774.115168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1774.240983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1774.533518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.557930] LNet: 83369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.565831] LNet: 83369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1774.570274] LNetError: 83369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.578229] LNetError: 83369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1774.593302] LNet: Removed LNI 192.168.202.28@tcp [ 1774.621990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.740150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1774.778918] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1775.737575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1775.954223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1777.064412] LNet: Removed LNI 192.168.202.28@tcp [ 1777.101177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.233804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1777.286196] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1778.172507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1778.237564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1778.393875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.434265] LNet: Removed LNI 192.168.202.28@tcp [ 1779.468613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.582794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1779.623588] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1780.483455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1780.706386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.800290] LNet: Removed LNI 192.168.202.28@tcp [ 1781.832425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.902107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1781.934899] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1782.844672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1782.968261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1783.180795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.200181] LNet: 83955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.205340] LNet: 83955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1783.211303] LNetError: 83955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.217412] LNetError: 83955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1783.227932] LNet: Removed LNI 192.168.202.28@tcp [ 1783.247945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.307686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1783.328623] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1784.182209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1784.612025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.703452] LNet: Removed LNI 192.168.202.28@tcp [ 1785.763143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.838659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1785.867114] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1786.662096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1786.830569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1787.144299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.208865] LNet: Removed LNI 192.168.202.28@tcp [ 1788.273044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.381902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1788.441883] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1789.053928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1790.559467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1791.595309] LNet: Removed LNI 192.168.202.28@tcp [ 1791.624418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.695815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1791.727249] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1792.452990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1792.545282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1792.564171] LNet: Removed LNI 192.168.202.28@tcp [ 1792.582798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.625780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1792.644026] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1793.471646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1793.551338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.585824] LNet: Removed LNI 192.168.202.28@tcp [ 1793.605904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.697226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1793.787968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.861698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.975614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1794.101524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.147416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.270889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1794.353419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.385655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.450713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1794.482930] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1795.493586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.542064] LNet: Removed LNI 192.168.202.28@tcp [ 1795.561990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.648708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1803.512996] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 16:23:56 (1787862236) [ 1803.614159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.629600] LNet: 85419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.632159] LNet: 85419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1803.643345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.691272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1803.701814] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 1805.124228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.150037] LNetError: 85563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.158152] LNetError: 85563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1805.169836] LNet: Removed LNI 192.168.202.28@tcp231 [ 1805.198994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.304965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1805.319748] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 1805.322680] LNet: 85571:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1805.714207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1805.749762] LNet: Removed LNI 192.168.202.28@tcp231 [ 1805.763750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1805.786241] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 1807.166204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1807.240346] LNet: Removed LNI 192.168.202.28@tcp231 [ 1807.267273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.352934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1807.382958] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/0] [ 1807.387281] LNet: 85723:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1808.308959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1808.415036] LNet: Removed LNI 192.168.202.28@tcp231 [ 1808.443728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1808.474388] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/0] [ 1809.709387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1809.776512] LNet: Removed LNI 192.168.202.28@tcp231 [ 1809.817575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.861392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1809.877163] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/60] [ 1809.879976] LNet: 85875:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1810.084275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1810.103576] LNet: Removed LNI 192.168.202.28@tcp231 [ 1810.116514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1810.128324] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/60] [ 1811.125689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.149085] LNet: Removed LNI 192.168.202.28@tcp231 [ 1811.161932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.225760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1811.245031] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 1811.634318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1811.690712] LNet: Removed LNI 192.168.202.28@tcp231 [ 1811.714722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1811.752117] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 1821.214319] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 16:24:14 (1787862254) [ 1821.289699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.315899] LNet: Removed LNI 192.168.202.28@tcp231 [ 1821.340211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.427359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1821.446859] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 1822.340645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1822.463333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1822.521849] LNet: Removed LNI 192.168.202.28@tcp232 [ 1822.585267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1822.660417] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 1824.321489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1824.397991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1824.449377] LNet: Removed LNI 192.168.202.28@tcp232 [ 1824.494670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1824.566546] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 1825.850840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1825.923269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1825.959737] LNet: Removed LNI 192.168.202.28@tcp232 [ 1826.034245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1826.086916] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 1827.461473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1827.494477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1827.509938] LNet: Removed LNI 192.168.202.28@tcp232 [ 1827.522078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1834.212878] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 16:24:27 (1787862267) [ 1836.324377] Key type .llcrypt unregistered [ 1836.341678] Key type ._llcrypt unregistered [ 1843.453640] Key type ._llcrypt registered [ 1843.455295] Key type .llcrypt registered [ 1843.739847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1843.762472] alg: No test for adler32 (adler32-zlib) [ 1844.888888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1844.934579] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1854.664906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1868.035157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.088710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1868.105781] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1869.347884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_interrupt -r 2 [ 1869.401680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_interrupt -r 2 [ 1869.475628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_interrupt -r 2 [ 1869.517082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_interrupt -r 2 [ 1869.549302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_interrupt -r 2 [ 1869.582985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_interrupt -r 2 [ 1869.635899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_interrupt -r 2 [ 1869.684040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_interrupt -r 2 [ 1869.712989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1869.725667] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1869.730097] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1870.018917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1870.077615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1870.109370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1870.156247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1870.201631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1870.271117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1870.312320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1870.353349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1870.368037] LNet: 88536:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1870.386141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1870.408531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1870.435672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1870.470868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1870.512624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1870.556683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1871.139711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_dropped -r 2 [ 1871.193688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_dropped -r 2 [ 1871.220735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_dropped -r 2 [ 1871.265561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_dropped -r 2 [ 1871.311835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_dropped -r 2 [ 1871.367353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_dropped -r 2 [ 1871.447828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_dropped -r 2 [ 1871.521312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_dropped -r 2 [ 1871.599748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1871.639334] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1871.917612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1871.943216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1871.966615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1871.987165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1872.008437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1872.035376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1872.064326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1872.096712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1872.120765] LNet: 88656:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1872.143730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1872.181934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1872.200193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1872.229800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1872.254886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1872.289128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1872.963577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_aborted -r 2 [ 1873.017822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_aborted -r 2 [ 1873.084846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_aborted -r 2 [ 1873.132895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_aborted -r 2 [ 1873.200490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_aborted -r 2 [ 1873.249769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_aborted -r 2 [ 1873.308558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_aborted -r 2 [ 1873.367768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_aborted -r 2 [ 1873.434778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1873.475581] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1873.802781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1873.829842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1873.853635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1873.884339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1873.921641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1873.958528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1873.986055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1874.013278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1874.027645] LNet: 88776:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1874.047342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1874.075635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1874.110585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1874.145903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1874.174791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1874.262983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1874.913385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_no_route -r 2 [ 1874.949274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_no_route -r 2 [ 1874.980704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_no_route -r 2 [ 1875.021977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_no_route -r 2 [ 1875.087739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_no_route -r 2 [ 1875.127510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_no_route -r 2 [ 1875.185456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_no_route -r 2 [ 1875.228502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_no_route -r 2 [ 1875.277428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1875.594446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1875.624766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1875.668714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1875.702405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1875.740454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1875.782451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1875.819507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1875.845509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1875.871579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1875.900301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1875.930774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1875.958235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1875.984846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1876.009324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1876.383235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_timeout -r 2 [ 1876.415311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_timeout -r 2 [ 1876.445529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_timeout -r 2 [ 1876.473884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_timeout -r 2 [ 1876.501219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_timeout -r 2 [ 1876.530857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_timeout -r 2 [ 1876.563460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_timeout -r 2 [ 1876.607929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_timeout -r 2 [ 1876.666965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1876.688596] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1876.692509] LNet: Skipped 1 previous similar message [ 1876.932552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1876.959329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1876.984708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1877.008872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1877.040447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1877.074575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1877.128439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1877.171922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1877.187448] LNet: 89017:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1877.192313] LNet: 89017:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1877.210108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1877.252503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1877.303895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1877.351687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1877.387959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1877.457285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1878.066429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_dropped -r 2 [ 1878.104904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e remote_dropped -r 2 [ 1878.130837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_dropped -r 2 [ 1878.154906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e remote_dropped -r 2 [ 1878.190673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e remote_dropped -r 2 [ 1878.212342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e remote_dropped -r 2 [ 1878.234616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e remote_dropped -r 2 [ 1878.270560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e remote_dropped -r 2 [ 1878.298295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1878.315627] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp [ 1878.492544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1878.524314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1878.545306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1878.561425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1878.575885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1878.592658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1878.607641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1878.626508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1878.648507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1878.667893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1878.689072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1878.710845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1878.729816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1878.779971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1884.852555] LNet: 89216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.858029] LNetError: 89216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.870225] LNet: Removed LNI 192.168.202.28@tcp [ 1884.880920] LNet: Removed LNI 192.168.202.28@tcp1 [ 1885.206691] Key type .llcrypt unregistered [ 1885.209183] Key type ._llcrypt unregistered [ 1905.394370] Key type ._llcrypt registered [ 1905.396018] Key type .llcrypt registered [ 1906.983780] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 16:25:39 (1787862339) [ 1907.354350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1907.369680] alg: No test for adler32 (adler32-zlib) [ 1908.314551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1908.360912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1908.384546] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1909.452111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1909.489454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.28@tcp [ 1909.512852] LNetError: 90197:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.28@tcp to recovery [ 1909.613743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.28@tcp [ 1909.710519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1915.701560] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 16:25:49 (1787862349) [ 1916.012653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1916.057505] LNet: 90786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.071467] LNetError: 90786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.086291] LNet: Removed LNI 192.168.202.28@tcp [ 1916.102421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.174929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1916.218621] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1916.916739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1916.957401] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1918.258145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.28@tcp [ 1918.298145] LNetError: 91052:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.28@tcp to recovery [ 1919.200317] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1919.203510] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1919.807643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.28@tcp [ 1929.141428] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 16:26:02 (1787862362) [ 1931.117618] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg228-server [ 1933.268271] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 16:26:05 (1787862365) [ 1935.824456] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg228-server [ 1938.032666] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 16:26:10 (1787862370) [ 1941.075238] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg228-server [ 1943.046982] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 16:26:16 (1787862376) [ 1943.420303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.452270] LNet: 92683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.483035] LNetError: 92683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.544601] LNet: Removed LNI 192.168.202.28@tcp [ 1944.549121] LNetError: 92683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.572279] LNet: Removed LNI 192.168.202.28@tcp1 [ 1944.633438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1944.740994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1944.768431] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1945.420395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1945.527811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1945.612736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1945.736482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1945.775428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1945.851806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1953.847426] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 16:26:27 (1787862387) [ 1953.988158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1954.049363] LNet: 93408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1954.061440] LNetError: 93408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1954.081929] LNet: Removed LNI 192.168.202.28@tcp [ 1954.105590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1954.154866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1954.210702] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1961.951375] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 16:26:35 (1787862395) [ 1962.113426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1962.184285] LNet: 94106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.198375] LNetError: 94106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1962.243109] LNet: Removed LNI 192.168.202.28@tcp [ 1962.301066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.392665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1962.409970] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1970.014389] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 16:26:42 (1787862402) [ 1971.740816] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1973.732705] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 16:26:46 (1787862406) [ 1974.387924] LNet: 95148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1974.391879] LNetError: 95148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1974.403075] LNet: Removed LNI 192.168.202.28@tcp [ 1975.190793] Key type .llcrypt unregistered [ 1975.192350] Key type ._llcrypt unregistered [ 1982.754155] Key type ._llcrypt registered [ 1982.758729] Key type .llcrypt registered [ 1982.942573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1982.969631] alg: No test for adler32 (adler32-zlib) [ 1983.922913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1983.959910] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1992.743543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1997.882690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.128@tcp [ 1998.415768] LNet: 95574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.426804] LNetError: 95574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1998.437586] LNet: Removed LNI 192.168.202.28@tcp [ 1998.865869] Key type .llcrypt unregistered [ 1998.869287] Key type ._llcrypt unregistered [ 2011.119142] Key type ._llcrypt registered [ 2011.120622] Key type .llcrypt registered [ 2012.425911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2013.842528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2015.520280] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 16:27:28 (1787862448) [ 2017.359524] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2019.190539] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 16:27:32 (1787862452) [ 2020.672681] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg228-server [ 2022.244478] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 16:27:35 (1787862455) [ 2024.260706] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg228-server [ 2025.470746] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 16:27:39 (1787862459) [ 2026.861332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2026.888420] alg: No test for adler32 (adler32-zlib) [ 2028.183628] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 2028.439870] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2030.088493] Key type lgssc registered [ 2031.315402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2059.381876] Key type lgssc unregistered [ 2059.743792] LNet: 98771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.754986] LNetError: 98771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2059.784703] LNet: Removed LNI 192.168.202.28@tcp [ 2060.490566] Key type .llcrypt unregistered [ 2060.493184] Key type ._llcrypt unregistered [ 2079.160875] Key type ._llcrypt registered [ 2079.165784] Key type .llcrypt registered [ 2080.850939] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 16:28:33 (1787862513) [ 2081.305308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2081.326927] alg: No test for adler32 (adler32-zlib) [ 2082.300974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2082.429871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2082.540103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2082.627169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2082.696315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2090.070279] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 16:28:43 (1787862523) [ 2090.305674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2090.372052] LNet: 100186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.405922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2090.475337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2090.518071] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2091.777678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2091.800687] LNet: 100330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.807620] LNetError: 100330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2091.825851] LNet: Removed LNI 192.168.202.28@tcp [ 2091.841821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2091.909871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2091.957195] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2092.745029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2092.782181] LNetError: 100358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.792679] LNet: Removed LNI 192.168.202.28@tcp [ 2092.804516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2092.837885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2092.854617] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2100.653922] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 16:28:53 (1787862533) [ 2100.970528] LNet: 100948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.975203] LNet: 100948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2100.980948] LNetError: 100948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2101.000149] LNet: Removed LNI 192.168.202.28@tcp [ 2101.664815] Key type .llcrypt unregistered [ 2101.675392] Key type ._llcrypt unregistered [ 2112.652216] Key type ._llcrypt registered [ 2112.654136] Key type .llcrypt registered [ 2112.872831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2112.906347] alg: No test for adler32 (adler32-zlib) [ 2114.007503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2114.132548] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2114.137208] LNet: *** cfs_fail_loc=e003, val=100*** [ 2115.869503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2115.894643] LNet: *** cfs_fail_loc=e003, val=100*** [ 2115.899983] LNet: Skipped 2 previous similar messages [ 2115.907562] Ping reply received with mapping: [ 2115.909517] found metadata reply NI[192.168.202.28@tcp] [ 2115.913689] 1c ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2115.917583] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2115.920511] 5a 5a 5a 5a 5a 5a 5a 5a [ 2116.134499] LNet: 101352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2116.141217] LNetError: 101352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2117.164866] LNet: Removed LNI 192.168.202.28@tcp [ 2118.499538] Key type .llcrypt unregistered [ 2118.501064] Key type ._llcrypt unregistered [ 2135.794723] Key type ._llcrypt registered [ 2135.800748] Key type .llcrypt registered [ 2137.226711] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 16:29:30 (1787862570) [ 2137.931471] Key type .llcrypt unregistered [ 2137.933552] Key type ._llcrypt unregistered [ 2145.950338] Key type ._llcrypt registered [ 2145.954611] Key type .llcrypt registered [ 2146.127834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2146.147888] alg: No test for adler32 (adler32-zlib) [ 2147.467827] LNetError: 102411:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2147.904696] Key type .llcrypt unregistered [ 2147.912347] Key type ._llcrypt unregistered [ 2153.208039] Key type ._llcrypt registered [ 2153.210351] Key type .llcrypt registered [ 2154.455971] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 16:29:47 (1787862587) [ 2155.239579] Key type .llcrypt unregistered [ 2155.241573] Key type ._llcrypt unregistered [ 2163.227281] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2163.231203] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2164.820453] Key type ._llcrypt registered [ 2164.823120] Key type .llcrypt registered [ 2165.105950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2165.135964] alg: No test for adler32 (adler32-zlib) [ 2166.276459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2166.388158] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2178.322752] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 16:30:11 (1787862611) [ 2178.406320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2178.451523] LNet: 104186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2178.471289] LNetError: 104186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2178.501580] LNet: Removed LNI 10.1.2.3@tcp [ 2178.539524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2179.525971] Key type .llcrypt unregistered [ 2179.533318] Key type ._llcrypt unregistered [ 2187.452760] Key type ._llcrypt registered [ 2187.456595] Key type .llcrypt registered [ 2187.703984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2187.750972] alg: No test for adler32 (adler32-zlib) [ 2188.628108] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2188.687412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2188.837294] LNet: 104486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2188.840978] LNetError: 104486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2189.864387] LNet: Removed LNI 192.168.202.28@tcp [ 2190.744548] Key type .llcrypt unregistered [ 2190.750617] Key type ._llcrypt unregistered [ 2204.221675] Key type ._llcrypt registered [ 2204.227720] Key type .llcrypt registered [ 2205.911580] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 16:30:38 (1787862638) [ 2206.601393] Key type .llcrypt unregistered [ 2206.603159] Key type ._llcrypt unregistered [ 2214.513123] Key type ._llcrypt registered [ 2214.522126] Key type .llcrypt registered [ 2214.796580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2214.815856] alg: No test for adler32 (adler32-zlib) [ 2223.183822] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 16:30:56 (1787862656) [ 2223.591301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2223.638584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2223.717821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2223.767457] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2224.436443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2224.470949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2224.518470] LNet: Removed LNI 192.168.202.28@tcp [ 2224.814909] LNet: 106246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2225.310644] Key type .llcrypt unregistered [ 2225.316182] Key type ._llcrypt unregistered [ 2238.390569] Key type ._llcrypt registered [ 2238.392463] Key type .llcrypt registered [ 2239.535041] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 16:31:12 (1787862672) [ 2239.998645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2240.005872] alg: No test for adler32 (adler32-zlib) [ 2240.959696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.028578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2241.062180] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2242.237519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2242.266305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.28@tcp [ 2242.284485] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2242.287584] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2242.290332] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2242, 0, 0 [ 2242.295213] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2242.297822] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2242.300989] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2242.303398] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2242, 0, 0 [ 2242.307277] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2242.309721] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2242.312610] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2242.315360] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2242, 0, 2292 [ 2242.319478] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2242.321827] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2242.327544] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2242.330864] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2242, 0, 2292 [ 2242.337538] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2250.948772] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 16:31:23 (1787862683) [ 2251.652646] LNet: 107788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2251.655769] LNetError: 107788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.666499] LNet: Removed LNI 192.168.202.28@tcp [ 2252.411184] Key type .llcrypt unregistered [ 2252.416532] Key type ._llcrypt unregistered [ 2261.086773] Key type ._llcrypt registered [ 2261.089435] Key type .llcrypt registered [ 2261.450216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2261.483590] alg: No test for adler32 (adler32-zlib) [ 2262.468561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2262.496927] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2272.751449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 2284.270969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.373938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2284.414060] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 2285.778588] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2286.117433] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2286.129597] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2286.137844] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2288.526984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2288.562629] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2300.788590] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 16:32:13 (1787862733) [ 2301.094798] LNet: 109299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.113088] LNetError: 109299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2301.135589] LNet: Removed LNI 192.168.202.28@tcp [ 2302.176137] LNet: Removed LNI 192.168.202.28@tcp1 [ 2302.183757] LNetError: 109299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2302.194581] LNetError: 109299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2302.225700] LNet: Removed LNI 10.1.2.3@tcp99 [ 2303.733946] Key type .llcrypt unregistered [ 2303.737484] Key type ._llcrypt unregistered [ 2314.116437] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2314.119392] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2317.241064] Key type ._llcrypt registered [ 2317.243900] Key type .llcrypt registered [ 2317.490460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2317.516332] alg: No test for adler32 (adler32-zlib) [ 2318.496954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2318.615941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2318.648645] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2319.600676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2319.629843] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2322.754262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2322.864734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2322.921787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2322.956936] LNetError: 110051:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2323.026942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2323.077815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2323.797215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2323.921393] LNet: 110071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2323.928167] LNetError: 110071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2323.935426] LNet: Removed LNI 10.1.2.3@tcp [ 2324.969292] LNet: Removed LNI 192.168.202.28@tcp [ 2325.982597] Key type .llcrypt unregistered [ 2325.990829] Key type ._llcrypt unregistered [ 2340.003398] Key type ._llcrypt registered [ 2340.009875] Key type .llcrypt registered [ 2341.808886] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 16:32:54 (1787862774) [ 2342.262900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2342.276446] alg: No test for adler32 (adler32-zlib) [ 2343.324852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.447418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2343.490642] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2344.306635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2344.413929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg228-client@tcp [ 2354.135237] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 16:33:06 (1787862786) [ 2354.267532] LNet: 111602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.279442] LNetError: 111602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2354.311763] LNet: Removed LNI 192.168.202.28@tcp [ 2355.612421] Key type .llcrypt unregistered [ 2355.618524] Key type ._llcrypt unregistered [ 2364.131187] Key type ._llcrypt registered [ 2364.135309] Key type .llcrypt registered [ 2364.339604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2364.356449] alg: No test for adler32 (adler32-zlib) [ 2365.344401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2365.408832] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2365.460324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2372.896700] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 16:33:25 (1787862805) [ 2375.080351] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2376.599876] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 16:33:29 (1787862809) [ 2378.207050] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2380.376211] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 16:33:33 (1787862813) [ 2380.694476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2380.735993] LNet: 113033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2380.749498] LNetError: 113033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2380.773954] LNet: Removed LNI 192.168.202.28@tcp [ 2380.809877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.903852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2380.936260] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2382.057401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2382.144542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2390.289642] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 16:33:43 (1787862823) [ 2390.417964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2390.442259] LNet: 113736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.448061] LNetError: 113736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2390.477418] LNet: Removed LNI 192.168.202.28@tcp [ 2390.511153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2390.613304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2390.689304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2391.567589] Key type .llcrypt unregistered [ 2391.574883] Key type ._llcrypt unregistered [ 2404.944741] Key type ._llcrypt registered [ 2404.947683] Key type .llcrypt registered [ 2406.167719] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 16:33:59 (1787862839) [ 2406.733204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2406.764282] alg: No test for adler32 (adler32-zlib) [ 2407.635201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2407.676720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2407.699435] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2408.310964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2408.365185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2415.003774] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 16:34:08 (1787862848) [ 2415.345805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2415.378601] LNet: 115302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2415.400178] LNetError: 115302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2415.415027] LNet: Removed LNI 192.168.202.28@tcp [ 2415.444933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2415.529580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2415.582914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2422.385619] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 16:34:15 (1787862855) [ 2422.807059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2422.845313] LNet: 115882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2422.868613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2422.939681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2423.554396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2423.618514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2423.660812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2424.080952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2424.101790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2424.220230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2424.257614] LNet: 115916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.290649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2424.367656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2424.846358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2424.913810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2425.018772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2425.719987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2425.765218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2431.561976] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 16:34:24 (1787862864) [ 2431.819925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2431.839939] LNet: 116516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2431.854831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2431.891288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2431.907397] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2432.372635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2432.384845] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2433.067537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2433.116226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2433.275150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2433.294189] LNetError: 116795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2433.301339] LNet: Removed LNI 192.168.202.28@tcp [ 2433.312925] LNet: Removed LNI 192.168.202.28@tcp2 [ 2433.331691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2433.393401] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2433.396822] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2433.400034] LNet: 116801:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2433.855321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2434.119400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2435.168262] LNet: Removed LNI 192.168.202.28@tcp [ 2435.173550] LNetError: 116838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2435.180362] LNetError: 116838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2435.205066] LNet: Removed LNI 192.168.202.28@tcp2 [ 2435.230544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2435.306924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2435.334115] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2435.918852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2435.937584] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2436.690807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2436.729110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2436.779778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2436.943653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2436.962985] LNet: 117123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2436.966888] LNet: 117123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2437.986076] LNet: Removed LNI 192.168.202.28@tcp [ 2437.991161] LNetError: 117123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2438.002478] LNetError: 117123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2438.016314] LNet: Removed LNI 192.168.202.28@tcp2 [ 2438.035730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2438.090813] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2438.096346] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2438.099690] LNet: 117129:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2438.719471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2438.930987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2438.949765] LNet: Removed LNI 192.168.202.28@tcp [ 2439.985277] LNet: Removed LNI 192.168.202.28@tcp2 [ 2440.023309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.081712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2440.112715] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2440.790714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2440.832319] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2441.868282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2441.944869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2442.177662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2442.200332] LNetError: 117445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2442.205972] LNetError: 117445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2443.232383] LNet: Removed LNI 192.168.202.28@tcp [ 2443.248882] LNet: Removed LNI 192.168.202.28@tcp2 [ 2443.328637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2443.464391] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2443.474235] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2443.478873] LNet: 117452:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2444.110258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2444.311836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2445.344627] LNet: Removed LNI 192.168.202.28@tcp [ 2445.363021] LNet: Removed LNI 192.168.202.28@tcp2 [ 2445.382721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2445.478372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2445.515304] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2446.184486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2446.202868] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2446.968750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2447.012744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2447.068694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2447.318991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2447.344677] LNet: 117773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2447.350295] LNet: 117773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2448.416186] LNet: Removed LNI 192.168.202.28@tcp [ 2449.447267] LNet: Removed LNI 192.168.202.28@tcp2 [ 2449.475073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2449.538251] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2449.547549] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2449.556108] LNet: 117779:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2449.957535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2450.168542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2450.205793] LNetError: 117817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2450.211834] LNetError: 117817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2450.217452] LNet: Removed LNI 192.168.202.28@tcp [ 2451.267641] LNet: Removed LNI 192.168.202.28@tcp2 [ 2451.324424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2451.395861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2451.416459] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2452.089688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2452.119828] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2453.050152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2453.081399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2453.146376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2453.561240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2454.624252] LNet: Removed LNI 192.168.202.28@tcp [ 2454.632796] LNet: Removed LNI 192.168.202.28@tcp2 [ 2454.698956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2454.755642] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2454.767611] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2454.773895] LNet: 118119:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2455.401530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2455.782470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2456.864345] LNet: Removed LNI 192.168.202.28@tcp [ 2456.873587] LNet: Removed LNI 192.168.202.28@tcp2 [ 2456.916389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2456.990327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2457.013921] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2457.777188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2457.791598] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2458.524498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2458.569970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2458.605930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2458.712742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2459.084834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2460.128471] LNet: Removed LNI 192.168.202.28@tcp [ 2461.165042] LNet: Removed LNI 192.168.202.28@tcp2 [ 2461.232102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2461.356456] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2461.364576] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2461.914840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2469.602348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2472.023765] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 16:35:04 (1787862904) [ 2472.183588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2472.231239] LNet: 119233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2472.237565] LNet: 119233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2472.246673] LNetError: 119233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2472.260246] LNetError: 119233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2472.265771] LNet: Removed LNI 192.168.202.28@tcp [ 2473.319897] LNet: Removed LNI 192.168.202.28@tcp2 [ 2473.381716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2473.762530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2473.800807] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2474.541336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2474.580767] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2475.772630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2475.790531] LNet: Removed LNI 10.1.2.3@tcp [ 2475.801561] LNet: Removed LNI 192.168.202.28@tcp [ 2475.822276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2475.926995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2475.956360] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2475.959598] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2476.577809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2476.612964] LNet: Removed LNI 192.168.202.28@tcp [ 2476.642820] LNet: Removed LNI 10.1.2.3@tcp [ 2483.830277] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 16:35:16 (1787862916) [ 2484.052218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2484.133750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2484.202373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2484.240461] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2484.990349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2485.047887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2485.090943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2485.133990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2485.178676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2485.206685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2485.233254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2485.266237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2485.359397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2485.470775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2485.521318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2485.733846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2492.255436] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 16:35:25 (1787862925) [ 2492.447185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2493.544430] LNet: Removed LNI 192.168.202.28@tcp [ 2493.566371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2493.766881] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2493.960358] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2493.968543] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2493.972888] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2496.497897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2496.552884] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2498.222901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2498.257300] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2502.028435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2502.105226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2502.232126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.28@tcp -m PUT -e local_timeout -r 1 [ 2502.384139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2502.502415] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2502.516696] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2502.927868] LNetError: 120912:0:(lib-move.c:4801:lnet_parse()) 192.168.202.28@tcp, src 192.168.202.28@tcp: Dropping PUT (error -108 looking up sender) [ 2503.969984] LNet: Removed LNI 10.1.2.3@tcp [ 2505.006756] LNet: Removed LNI 192.168.202.28@tcp [ 2505.666572] Key type .llcrypt unregistered [ 2505.669374] Key type ._llcrypt unregistered [ 2521.263234] Key type ._llcrypt registered [ 2521.267966] Key type .llcrypt registered [ 2522.602176] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 16:35:55 (1787862955) [ 2523.116888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2523.133659] alg: No test for adler32 (adler32-zlib) [ 2524.118591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2524.862555] LNet: 122227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2525.781668] Key type .llcrypt unregistered [ 2525.784953] Key type ._llcrypt unregistered [ 2533.993802] Key type ._llcrypt registered [ 2533.995987] Key type .llcrypt registered [ 2534.252459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2534.281298] alg: No test for adler32 (adler32-zlib) [ 2535.363922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2535.404313] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2545.855962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 2546.145720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2547.459030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.128@tcp [ 2547.681202] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp [ 2547.842393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2547.891059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2547.948127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2548.052258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2548.114435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2548.153699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2548.196517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2548.254226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2548.272433] LNet: 122682:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2548.310918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2548.354804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2548.388137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2548.460571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2548.518574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2548.561718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2549.847714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2550.512614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2550.600637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2550.699736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2550.735551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2550.794871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2550.871165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2550.932959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2551.004575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2551.042964] LNet: 122799:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2551.074894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2551.180451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2551.238909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2551.278049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2551.326748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2551.380452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2552.749915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.128@tcp [ 2553.338909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2553.472289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2553.683934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2553.796822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2553.860738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2553.895615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2553.941037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2554.010626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2554.060480] LNet: 122917:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2554.096963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2554.172823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2554.264667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2554.397479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2554.513724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2554.631892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2555.968327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2556.341912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2556.386509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2556.423984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2556.508865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2556.571237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2556.601806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2556.653679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2556.682689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2556.701334] LNet: 123034:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2556.719081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2556.749782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2556.797767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2556.823847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2556.884539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2556.947975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2557.456399] LNet: 123084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2557.463602] LNetError: 123084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2557.483813] LNet: Removed LNI 192.168.202.28@tcp [ 2558.251606] Key type .llcrypt unregistered [ 2558.252880] Key type ._llcrypt unregistered [ 2583.260898] Key type ._llcrypt registered [ 2583.265628] Key type .llcrypt registered [ 2585.424699] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 16:36:57 (1787863017) [ 2586.343585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2586.386774] alg: No test for adler32 (adler32-zlib) [ 2587.422908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2588.112892] LNet: 123973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2589.138324] Key type .llcrypt unregistered [ 2589.141503] Key type ._llcrypt unregistered [ 2599.642108] Key type ._llcrypt registered [ 2599.643581] Key type .llcrypt registered [ 2600.029582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2600.046281] alg: No test for adler32 (adler32-zlib) [ 2601.158174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2601.197375] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2611.232828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 2621.835114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2621.909348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2621.991711] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 2623.468362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2624.336972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.128@tcp [ 2624.678322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2624.717968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2624.764692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2624.802119] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp [ 2624.833773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2624.899729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2624.957841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2625.037443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2625.086173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2625.131275] LNet: 124631:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2625.163956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2625.265390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2625.296684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2625.345916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2625.377994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2625.406896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2626.233952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2626.476899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2626.514793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2626.551666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2626.601561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2626.662710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2626.701966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2626.734661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2626.771139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2626.796401] LNet: 124748:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2626.817647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2626.882226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2626.935500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2626.972150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2627.022728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2627.094786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2627.852380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.128@tcp [ 2628.079854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2628.120663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2628.166645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2628.187907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2628.215894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2628.243513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2628.282699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2628.329835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2628.374911] LNet: 124865:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2628.411422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2628.451992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2628.490722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2628.536132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2628.571622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2628.610773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2629.434237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2629.744293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2629.799864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2629.869544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2629.952089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2629.998221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2630.034915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2630.104320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2630.179802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2630.224159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2630.264609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2630.304533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2630.331485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2630.370892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2630.409888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2636.238700] LNet: 125067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2636.255765] LNetError: 125067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2636.282548] LNet: Removed LNI 192.168.202.28@tcp [ 2636.304719] LNet: Removed LNI 192.168.202.28@tcp1 [ 2636.936138] Key type .llcrypt unregistered [ 2636.937815] Key type ._llcrypt unregistered [ 2658.581871] Key type ._llcrypt registered [ 2658.584211] Key type .llcrypt registered [ 2660.118065] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 16:38:12 (1787863092) [ 2660.690695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2660.704518] alg: No test for adler32 (adler32-zlib) [ 2661.668810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2661.718892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2661.739820] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2662.464377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2662.482040] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2663.528227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2663.569380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2664.610706] LNet: Removed LNI 192.168.202.28@tcp [ 2672.069330] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 16:38:25 (1787863105) [ 2673.802344] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2675.809791] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 16:38:28 (1787863108) [ 2684.327395] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 16:38:37 (1787863117) [ 2692.985182] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 16:38:45 (1787863125) [ 2701.959804] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 16:38:54 (1787863134) [ 2708.892747] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 16:39:02 (1787863142) [ 2717.186531] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 16:39:10 (1787863150) [ 2724.558325] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 16:39:17 (1787863157) [ 2733.913690] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 16:39:26 (1787863166) [ 2743.610918] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 16:39:36 (1787863176) [ 2753.196204] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 16:39:45 (1787863185) [ 2761.483192] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 16:39:54 (1787863194) [ 2770.236321] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 16:40:03 (1787863203) [ 2779.174914] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 16:40:11 (1787863211) [ 2787.614430] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 16:40:20 (1787863220) [ 2795.827376] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 16:40:28 (1787863228) [ 2806.177458] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 16:40:39 (1787863239) [ 2815.287806] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 16:40:48 (1787863248) [ 2823.964199] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 16:40:57 (1787863257) [ 2824.304894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2824.334317] LNet: 136769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2824.348409] LNetError: 136769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2825.386933] LNet: Removed LNI 192.168.202.28@tcp2 [ 2825.438068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2825.605700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2825.642165] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2827.372807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2827.395841] LNet: 136912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2827.400678] LNetError: 136912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2827.415268] LNet: Removed LNI 192.168.202.28@tcp [ 2827.446384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2827.511158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2827.577110] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2836.242912] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 16:41:08 (1787863268) [ 2845.447293] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 16:41:17 (1787863277) [ 2856.145354] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 16:41:28 (1787863288) [ 2865.351584] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 16:41:38 (1787863298) [ 2873.234190] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 16:41:46 (1787863306) [ 2880.276493] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 16:41:53 (1787863313) [ 2886.610255] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 16:42:00 (1787863320) [ 2895.162537] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 16:42:07 (1787863327) [ 2906.100466] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 16:42:18 (1787863338) [ 2915.544540] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 16:42:27 (1787863347) [ 2923.271124] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 16:42:36 (1787863356) [ 2930.510050] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 16:42:43 (1787863363) [ 2939.396847] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2674 sec ========== 16:42:52 (1787863372) [ 2940.086648] LNet: 144390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2940.096184] LNetError: 144390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2940.110874] LNet: Removed LNI 192.168.202.28@tcp [ 2941.139800] Key type .llcrypt unregistered [ 2941.141348] Key type ._llcrypt unregistered [ 2952.120469] Key type ._llcrypt registered [ 2952.133146] Key type .llcrypt registered [ 2952.332226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2952.343043] alg: No test for adler32 (adler32-zlib) [ 2953.388687] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 2953.661322] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2955.360662] Key type lgssc registered [ 2957.534355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3029.605303] Lustre: Mounted lustre-client - version 2.17.57_120_gb73d7ca [ 3034.986776] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3042.969183] Lustre: Unmounted lustre-client [ 3068.533781] Key type lgssc unregistered [ 3068.818937] LNet: 147355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3068.825743] LNetError: 147355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3068.843087] LNet: Removed LNI 192.168.202.28@tcp [ 3069.833313] Key type .llcrypt unregistered [ 3069.837049] Key type ._llcrypt unregistered