[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 527049973 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003098] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008033] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010148] LSM: Security Framework initializing [ 0.011079] Yama: becoming mindful. [ 0.012028] SELinux: Initializing. [ 0.013016] *** VALIDATE selinux *** [ 0.022079] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027210] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028202] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030151] *** VALIDATE tmpfs *** [ 0.032173] *** VALIDATE proc *** [ 0.033295] *** VALIDATE cgroup *** [ 0.035001] *** VALIDATE cgroup2 *** [ 0.036284] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039032] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.042718] debug: unmapping init [mem 0xffffffff8ee59000-0xffffffff8ee60fff] [ 0.044186] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045754] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046035] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051019] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053371] rcu: Hierarchical SRCU implementation. [ 0.055530] smp: Bringing up secondary CPUs ... [ 0.056626] x86: Booting SMP configuration: [ 0.057040] .... node #0, CPUs: #1 #2 #3 [ 0.060498] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.126087] node 0 deferred pages initialised in 61ms [ 0.128129] devtmpfs: initialized [ 0.129262] x86/mm: Memory block size: 128MB [ 0.132050] gcov: version magic: 0x41383552 [ 0.135308] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137308] pinctrl core: initialized pinctrl subsystem [ 0.138341] [ 0.138899] ************************************************************* [ 0.139017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140016] ** ** [ 0.141017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.142014] ** ** [ 0.143016] ** This means that this kernel is built to expose internal ** [ 0.144016] ** IOMMU data structures, which may compromise security on ** [ 0.145012] ** your system. ** [ 0.146009] ** ** [ 0.147011] ** If you see this message and you are not debugging the ** [ 0.148011] ** kernel, report this immediately to your vendor! ** [ 0.149012] ** ** [ 0.150012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151013] ************************************************************* [ 0.152709] NET: Registered protocol family 16 [ 0.153429] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.154091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.155072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.157096] cpuidle: using governor menu [ 0.159762] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.161416] PCI: Using configuration type 1 for base access [ 0.163125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.173122] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.174033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.176125] cryptd: max_cpu_qlen set to 1000 [ 0.179263] ACPI: Added _OSI(Module Device) [ 0.180013] ACPI: Added _OSI(Processor Device) [ 0.181013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.182015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.186922] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.188531] ACPI: Interpreter enabled [ 0.189066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.190011] ACPI: Using IOAPIC for interrupt routing [ 0.191133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.192460] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.202120] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.203038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.204015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.205092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.207451] acpiphp: Slot [2] registered [ 0.208170] acpiphp: Slot [5] registered [ 0.209136] acpiphp: Slot [6] registered [ 0.210144] acpiphp: Slot [3] registered [ 0.211106] acpiphp: Slot [4] registered [ 0.212103] acpiphp: Slot [7] registered [ 0.213097] acpiphp: Slot [8] registered [ 0.214303] acpiphp: Slot [9] registered [ 0.215121] acpiphp: Slot [10] registered [ 0.216128] acpiphp: Slot [11] registered [ 0.217107] acpiphp: Slot [12] registered [ 0.218111] acpiphp: Slot [13] registered [ 0.219126] acpiphp: Slot [14] registered [ 0.220109] acpiphp: Slot [15] registered [ 0.221142] acpiphp: Slot [16] registered [ 0.222112] acpiphp: Slot [17] registered [ 0.223113] acpiphp: Slot [18] registered [ 0.224107] acpiphp: Slot [19] registered [ 0.225102] acpiphp: Slot [20] registered [ 0.226100] acpiphp: Slot [21] registered [ 0.227131] acpiphp: Slot [22] registered [ 0.228112] acpiphp: Slot [23] registered [ 0.229164] acpiphp: Slot [24] registered [ 0.230115] acpiphp: Slot [25] registered [ 0.231105] acpiphp: Slot [26] registered [ 0.232092] acpiphp: Slot [27] registered [ 0.233143] acpiphp: Slot [28] registered [ 0.234117] acpiphp: Slot [29] registered [ 0.235090] acpiphp: Slot [30] registered [ 0.236161] acpiphp: Slot [31] registered [ 0.237066] PCI host bridge to bus 0000:00 [ 0.238020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.239026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.240032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.241034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.242023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.243094] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.245244] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.248088] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.251395] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.259015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.263060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.266019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.268020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.271019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.274056] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.278216] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.282106] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.285878] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.289985] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.298015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.303016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.307927] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.316017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.322020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.335015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.348359] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.358018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.369018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.401022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.417550] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.419419] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.422340] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.424397] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.426232] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.430192] iommu: Default domain type: Passthrough [ 0.432276] SCSI subsystem initialized [ 0.434160] ACPI: bus type USB registered [ 0.435101] usbcore: registered new interface driver usbfs [ 0.437098] usbcore: registered new interface driver hub [ 0.439094] usbcore: registered new device driver usb [ 0.441196] pps_core: LinuxPPS API ver. 1 registered [ 0.443019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.446064] PTP clock support registered [ 0.448070] EDAC MC: Ver: 3.0.0 [ 0.449643] PCI: Using ACPI for IRQ routing [ 0.450490] NetLabel: Initializing [ 0.451000] NetLabel: domain hash size = 128 [ 0.454014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.456077] NetLabel: unlabeled traffic allowed by default [ 0.458231] vgaarb: loaded [ 0.460296] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.461014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.467196] clocksource: Switched to clocksource kvm-clock [ 0.584334] VFS: Disk quotas dquot_6.6.0 [ 0.585400] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.588103] *** VALIDATE ramfs *** [ 0.589132] *** VALIDATE hugetlbfs *** [ 0.590088] pnp: PnP ACPI init [ 0.592564] pnp: PnP ACPI: found 6 devices [ 0.608635] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.612081] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.614360] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.616514] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.618527] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.620865] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.623579] NET: Registered protocol family 2 [ 0.626352] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.631928] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.635994] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.641651] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.645145] TCP: Hash tables configured (established 65536 bind 65536) [ 0.648115] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.651286] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.653745] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.656555] NET: Registered protocol family 1 [ 0.660146] RPC: Registered named UNIX socket transport module. [ 0.662558] RPC: Registered udp transport module. [ 0.664410] RPC: Registered tcp transport module. [ 0.666248] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.668798] NET: Registered protocol family 44 [ 0.670488] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.673035] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.675310] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.677627] PCI: CLS 0 bytes, default 64 [ 0.679479] Unpacking initramfs... [ 1.993710] debug: unmapping init [mem 0xffff8b8ffcc64000-0xffff8b8ffffcffff] [ 1.997540] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.999475] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.002217] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.449892] Initialise system trusted keyrings [ 2.451059] Key type blacklist registered [ 2.453211] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.461813] zbud: loaded [ 2.464661] *** VALIDATE nfs *** [ 2.465699] *** VALIDATE nfs4 *** [ 2.467176] pstore: using deflate compression [ 2.470921] Platform Keyring initialized [ 2.550442] NET: Registered protocol family 38 [ 2.551510] Key type asymmetric registered [ 2.552883] Asymmetric key parser 'x509' registered [ 2.554138] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.555883] io scheduler mq-deadline registered [ 2.556953] io scheduler kyber registered [ 2.557897] io scheduler bfq registered [ 2.559300] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.561161] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.563092] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.565213] ACPI: Power Button [PWRF] [ 2.568543] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.572572] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.579840] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.604213] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.632491] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.637035] Non-volatile memory driver v1.3 [ 2.638519] Linux agpgart interface v0.103 [ 2.665849] virtio_blk virtio1: [vda] 150640 512-byte logical blocks (77.1 MB/73.6 MiB) [ 2.668682] vda: detected capacity change from 0 to 77127680 [ 2.685764] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.688662] vdb: detected capacity change from 0 to 1073741824 [ 2.699673] libphy: Fixed MDIO Bus: probed [ 2.707453] usbcore: registered new interface driver usbserial_generic [ 2.709393] usbserial: USB Serial support registered for generic [ 2.711130] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.714941] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.716427] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.718629] mousedev: PS/2 mouse device common for all mice [ 2.721835] rtc_cmos 00:05: RTC can wake from S4 [ 2.724354] rtc_cmos 00:05: registered as rtc0 [ 2.725985] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.728353] intel_pstate: CPU model not supported [ 2.730636] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.733901] hid: raw HID events driver (C) Jiri Kosina [ 2.738178] usbcore: registered new interface driver usbhid [ 2.739471] usbhid: USB HID core driver [ 2.740683] drop_monitor: Initializing network drop monitor service [ 2.741028] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.742311] Initializing XFRM netlink socket [ 2.746278] NET: Registered protocol family 10 [ 2.749434] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.749506] Segment Routing with IPv6 [ 2.753986] NET: Registered protocol family 17 [ 2.755846] mpls_gso: MPLS GSO support [ 2.761262] RAS: Correctable Errors collector initialized. [ 2.762890] AVX version of gcm_enc/dec engaged. [ 2.764134] AES CTR mode by8 optimization enabled [ 2.839400] sched_clock: Marking stable (2839366548, 0)->(3744139570, -904773022) [ 2.846416] registered taskstats version 1 [ 2.848508] Loading compiled-in X.509 certificates [ 2.850664] zswap: loaded using pool lzo/zbud [ 2.873147] Key type big_key registered [ 2.884430] Key type encrypted registered [ 2.886125] ima: No TPM chip found, activating TPM-bypass! [ 2.888594] ima: Allocated hash algorithm: sha1 [ 2.890147] ima: No architecture policies found [ 2.892547] evm: Initialising EVM extended attributes: [ 2.894872] evm: security.selinux [ 2.896226] evm: security.ima [ 2.897183] evm: security.capability [ 2.898111] evm: HMAC attrs: 0x1 [ 2.899587] rtc_cmos 00:05: setting system clock to 2026-09-30 03:56:53 UTC (1790740613) [ 2.903578] debug: unmapping init [mem 0xffffffff8fe03000-0xffffffff8fffffff] [ 2.905986] debug: unmapping init [mem 0xffffffff8eb82000-0xffffffff8ee58fff] [ 2.912099] Write protecting the kernel read-only data: 28672k [ 2.914526] debug: unmapping init [mem 0xffffffff8d203000-0xffffffff8d3fffff] [ 2.917234] debug: unmapping init [mem 0xffffffff8db14000-0xffffffff8dbfffff] [ 2.946445] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.954223] systemd[1]: Detected virtualization kvm. [ 2.956209] systemd[1]: Detected architecture x86-64. [ 2.958319] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.984316] systemd[1]: No hostname configured. [ 2.985776] systemd[1]: Set hostname to . [ 2.986953] random: systemd: uninitialized urandom read (16 bytes read) [ 2.988587] systemd[1]: Initializing machine ID from random generator. [ 3.115129] random: systemd: uninitialized urandom read (16 bytes read) [ 3.118583] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.122929] random: systemd: uninitialized urandom read (16 bytes read) [ 3.124831] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.128888] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.665150] device-mapper: uevent: version 1.0.3 [ 3.667139] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.367341] scsi host0: ata_piix [ 4.388410] virtio_net virtio0 ens2: renamed from eth0 [ 4.396523] scsi host1: ata_piix [ 4.398577] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.401304] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.830657] dracut-initqueue[595]: RTNETLINK answers: File exists [ 9.435061] random: crng init done [ 9.436473] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.760862] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.906313] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.149914] SELinux: Disabled at runtime. [ 11.214456] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.222803] systemd[1]: Detected virtualization kvm. [ 11.224744] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.713701] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.717284] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.725327] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.730383] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.734380] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.746377] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.765599] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapp[ 11.873819] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS er. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.202961] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.516875] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.532178] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.641686] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.655985] EDAC sbridge: Ver: 1.1.2 [ 13.878906] Key type dns_resolver registered [ 14.222252] NFS: Registering the id_resolver key type [ 14.224835] Key type id_resolver registered [ 14.227389] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg228-client login: [ 87.863017] hrtimer: interrupt took 5005970 ns [ 88.586731] libcfs: loading out-of-tree module taints kernel. [ 88.704591] Key type ._llcrypt registered [ 88.707022] Key type .llcrypt registered [ 89.805436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 89.819736] alg: No test for adler32 (adler32-zlib) [ 91.504431] Lustre: Lustre: Build Version: 2.17.59_3_g21e240e [ 92.433307] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 94.223540] Key type lgssc registered [ 96.380976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 237.026097] Lustre: Mounted lustre-client - version 2.17.59_3_g21e240e [ 242.786104] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.520391] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing check_logdir /tmp/testlogs/ [ 262.624458] Lustre: lustre-OST0000-osc-ffff8b9050e47800: disconnect after 23s idle [ 262.861330] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing yml_node [ 267.710128] Lustre: DEBUG MARKER: Client: 2.17.59.3 [ 270.528056] Lustre: DEBUG MARKER: MDS: 2.17.59.3 [ 273.450256] Lustre: DEBUG MARKER: OSS: 2.17.59.3 [ 275.260893] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 30 00:01:24 EDT 2026 [ 293.036596] Lustre: Unmounted lustre-client [ 318.876298] Key type lgssc unregistered [ 319.195872] LNet: 8420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.202996] LNetError: 8420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 319.235026] LNet: Removed LNI 192.168.202.28@tcp [ 320.009320] Key type .llcrypt unregistered [ 320.012437] Key type ._llcrypt unregistered [ 350.487728] Key type ._llcrypt registered [ 350.491389] Key type .llcrypt registered [ 350.955363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 350.998157] alg: No test for adler32 (adler32-zlib) [ 351.983512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 352.070868] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 352.091339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 357.451236] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 357.534666] LNet: 9308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.538961] LNetError: 9308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 357.553658] LNet: Removed LNI 192.168.202.28@tcp [ 358.266554] Key type .llcrypt unregistered [ 358.270589] Key type ._llcrypt unregistered [ 367.102443] Key type ._llcrypt registered [ 367.107882] Key type .llcrypt registered [ 368.568240] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 00:02:57 (1790740977) [ 369.028963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 369.055995] alg: No test for adler32 (adler32-zlib) [ 369.972740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.032555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.044563] LNet: 9725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.057482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.107132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 370.123859] LNet: 9732:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 377.151686] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 00:03:06 (1790740986) [ 377.330342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.369347] LNet: 10306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.428445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.540838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 385.626771] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 00:03:14 (1790740994) [ 386.061557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.109973] LNet: 10897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.131967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.219719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 393.548956] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 00:03:22 (1790741002) [ 394.103838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.174474] LNet: 11483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.191542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.299151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 402.065868] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 00:03:31 (1790741011) [ 402.408906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.445358] LNet: 12068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.480686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.604882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 402.988382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 410.157931] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 00:03:39 (1790741019) [ 410.259755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.296710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.396628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 410.946244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 411.591566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 419.741282] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 00:03:48 (1790741028) [ 420.031461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.069850] LNet: 13254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.073830] LNet: 13254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 420.105085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.212241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 427.792629] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 00:03:56 (1790741036) [ 428.123575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.184407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.285873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 428.345247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 428.817887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 428.859341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 429.453375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 429.497371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 430.139157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 430.202981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 430.845401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 430.912829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 431.515719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 431.565801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 432.136904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 432.175516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 432.606782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 432.664692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 433.041790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 433.069665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 433.741038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 433.798432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 434.543285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 434.597864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 435.362596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 435.440396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 435.824112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 435.864731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 444.486609] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 00:04:13 (1790741053) [ 444.840979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.899741] LNet: 14521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 444.911923] LNet: 14521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 444.956321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.111811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 445.152204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 453.447284] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 00:04:21 (1790741061) [ 453.943451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.046642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.194271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 454.274048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 465.794599] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 00:04:33 (1790741073) [ 465.981476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.102273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.311608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 466.365039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 476.798783] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 00:04:45 (1790741085) [ 476.983323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.042028] LNet: 16278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 477.045489] LNet: 16278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 477.115206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.224994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 477.312782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 486.815424] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 00:04:55 (1790741095) [ 487.002444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.081068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.300286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 487.378561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 493.999712] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 00:05:03 (1790741103) [ 494.337962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.386441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.512717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 494.569393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 501.999122] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 00:05:10 (1790741110) [ 502.326436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.407741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.553685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 503.089834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 509.992931] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 00:05:18 (1790741118) [ 510.251626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.295276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.547692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.847835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 518.342355] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 00:05:27 (1790741127) [ 518.773114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.834407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.991945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 519.252982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 528.520126] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 00:05:37 (1790741137) [ 528.888041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.947939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.685833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 530.020098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 538.560640] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 00:05:47 (1790741147) [ 538.744438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.819969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.957930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 539.024587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 547.255585] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 00:05:55 (1790741155) [ 547.411198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.459739] LNet: 21107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 547.469567] LNet: 21107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 547.507828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.669082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 547.731315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 556.818652] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 00:06:05 (1790741165) [ 557.067771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.100738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.179831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 565.213467] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 00:06:14 (1790741174) [ 565.326976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.407418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.518761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 572.756670] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 00:06:22 (1790741182) [ 572.997600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.088147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.173670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 573.731457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 581.606791] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 00:06:30 (1790741190) [ 581.917285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.976637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.068186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 582.112888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 589.756742] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 00:06:38 (1790741198) [ 589.904784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.976127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.114150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 590.151285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 598.609296] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 00:06:47 (1790741207) [ 599.477775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.573335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.739228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 599.814481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 607.573682] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 00:06:56 (1790741216) [ 608.124105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.194560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.294826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 608.360901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 617.671837] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 00:07:06 (1790741226) [ 617.816058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.928888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.068589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 618.144926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 618.164351] LNetError: 25840:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 618.259658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 618.314217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 624.894604] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 00:07:14 (1790741234) [ 625.009104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.071510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.228087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.mlkBXzuqYz [ 625.324931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.mlkBXzuqYz [ 625.380699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.mlkBXzuqYz [ 632.110991] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 00:07:21 (1790741241) [ 632.591660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.640729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.744094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 632.820832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 632.926556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 640.873703] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 00:07:29 (1790741249) [ 641.363421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.421147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.548476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 641.592899] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 642.789542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 643.913943] Lustre: Lustre: Build Version: 2.17.59_3_g21e240e [ 645.904599] Key type lgssc registered [ 647.753804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 652.095516] LNetError: 27601:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.202.28@tcp99 has no available nets [ 654.376404] Key type lgssc unregistered [ 654.834233] LNetError: 28717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.847366] LNet: Removed LNI 192.168.202.28@tcp [ 655.398587] Key type .llcrypt unregistered [ 655.403885] Key type ._llcrypt unregistered [ 662.371298] Key type ._llcrypt registered [ 662.378743] Key type .llcrypt registered [ 664.031484] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 00:07:52 (1790741272) [ 664.570619] Key type .llcrypt unregistered [ 664.572662] Key type ._llcrypt unregistered [ 671.839147] Key type ._llcrypt registered [ 671.841320] Key type .llcrypt registered [ 672.075072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 672.115938] alg: No test for adler32 (adler32-zlib) [ 673.071632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.163328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 673.185337] LNet: 29640:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 673.194127] LNet: 29640:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 673.201659] LNet: 29640:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 673.214260] LNet: Message forwarding will be enabled in 2s [ 676.320407] LNet: Message forwarding enabled [ 676.746487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 676.804156] LNet: Message forwarding will stop in 2s to 6s [ 680.427495] LNet: Message forwarding disabled [ 681.364370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 681.387894] LNet: 29738:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 681.404445] LNet: 29738:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 681.416789] LNet: 29738:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 681.433767] LNet: Message forwarding will be enabled in 2s [ 681.562283] LNet: 29751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.119416] Key type .llcrypt unregistered [ 682.120689] Key type ._llcrypt unregistered [ 694.244693] Key type ._llcrypt registered [ 694.247927] Key type .llcrypt registered [ 695.559590] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 00:08:24 (1790741304) [ 696.065906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 696.091382] alg: No test for adler32 (adler32-zlib) [ 697.077280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.143612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 704.201712] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 00:08:33 (1790741313) [ 704.285586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.332324] LNet: 31147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.363655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.440087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 704.482557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 704.534726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 704.564870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 704.600953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 704.647614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 704.717435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 704.751297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 704.780152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 704.822914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 704.888878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 704.952868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 705.013321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 705.094720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 705.156443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 705.214445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 705.270830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 705.368568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 705.466287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 705.513412] LNetError: 31198:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 712.190622] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 00:08:41 (1790741321) [ 712.725619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.769767] LNet: 31770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.800338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.949331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 719.218673] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 00:08:48 (1790741328) [ 721.366344] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg228-server [ 723.302823] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 00:08:52 (1790741332) [ 723.572067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.620672] LNet: 32701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.672472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.744661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 723.770343] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 724.759266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[29-33]@tcp [ 724.790265] LNet: 32834:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 724.855067] LNetError: 32705:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.29@tcp is being used as a gateway but routing feature is not turned on [ 724.895060] LNetError: 32705:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.29@tcp has gone from up to down [ 731.825627] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 00:09:01 (1790741341) [ 733.918046] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg228-server [ 736.065371] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 00:09:04 (1790741344) [ 736.273111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.294289] LNet: 33769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.301644] LNetError: 33769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.362132] LNet: Removed LNI 192.168.202.28@tcp [ 736.388161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.453030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 736.498438] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 737.223453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[29-34/2]@tcp [ 737.259410] LNet: 33888:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 737.274805] LNet: 33888:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 737.314298] LNetError: 33775:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.31@tcp is being used as a gateway but routing feature is not turned on [ 737.318905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[29-34/2]@tcp [ 737.334064] LNetError: 33775:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 737.334091] LNetError: 33775:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.31@tcp has gone from up to down [ 737.334096] LNetError: 33775:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 744.481421] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 00:09:13 (1790741353) [ 744.895838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 744.929552] LNet: 34482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.936535] LNetError: 34482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.971461] LNet: Removed LNI 192.168.202.28@tcp [ 746.012606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.124354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 746.198300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.261325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.362514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 746.429917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.478643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.632714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 746.717323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.756599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.843485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 746.999360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.063926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.133425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 747.277950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.322236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.439428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 747.514400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.579405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.668168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 747.759436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.810627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.906526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 747.975323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.006873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.049576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 748.116380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.154526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.213675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 748.287355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.324216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.365524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 748.416734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.455614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.532376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 755.916590] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 00:09:24 (1790741364) [ 758.119412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg228-server [ 760.447487] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 00:09:29 (1790741369) [ 763.056537] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg228-server [ 765.082383] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 00:09:34 (1790741374) [ 765.796732] LNet: 35883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.818487] LNet: 35883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 766.898282] Key type .llcrypt unregistered [ 766.905800] Key type ._llcrypt unregistered [ 774.476389] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 774.778524] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 774.789461] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 774.808728] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 777.416069] Key type ._llcrypt registered [ 777.417749] Key type .llcrypt registered [ 777.654444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 777.668673] alg: No test for adler32 (adler32-zlib) [ 778.605840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.665384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.709208] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 779.387858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 779.405384] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 781.603541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 781.630397] LNet: Removed LNI 10.1.2.3@tcp [ 781.790081] LNet: 36596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.797879] LNetError: 36596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.818550] LNet: Removed LNI 192.168.202.28@tcp [ 782.545782] Key type .llcrypt unregistered [ 782.548823] Key type ._llcrypt unregistered [ 795.785613] Key type ._llcrypt registered [ 795.789255] Key type .llcrypt registered [ 796.972945] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 00:10:06 (1790741406) [ 797.513412] Key type .llcrypt unregistered [ 797.515356] Key type ._llcrypt unregistered [ 803.135192] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 803.523115] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.534780] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 803.541790] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 806.206344] Key type ._llcrypt registered [ 806.208736] Key type .llcrypt registered [ 806.431041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 806.458411] alg: No test for adler32 (adler32-zlib) [ 807.389707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.439930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 807.476255] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 808.287962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 808.314850] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 810.846743] LNet: 38087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.852610] LNetError: 38087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 810.857522] LNet: Removed LNI 10.1.2.3@tcp [ 810.870711] LNet: Removed LNI 192.168.202.28@tcp [ 811.367729] Key type .llcrypt unregistered [ 811.372514] Key type ._llcrypt unregistered [ 822.669524] Key type ._llcrypt registered [ 822.671313] Key type .llcrypt registered [ 824.135530] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 00:10:33 (1790741433) [ 824.780284] Key type .llcrypt unregistered [ 824.782389] Key type ._llcrypt unregistered [ 832.374894] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 832.586256] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 832.590149] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 832.598752] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 835.204396] Key type ._llcrypt registered [ 835.206829] Key type .llcrypt registered [ 835.425788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 835.443300] alg: No test for adler32 (adler32-zlib) [ 836.522032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.599492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 836.626044] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 838.668987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 838.695498] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 840.690977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 840.708546] LNet: Removed LNI 10.1.2.3@tcp [ 840.741368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 840.797331] LNet: Removed LNI 10.1.2.31@tcp [ 840.843597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 840.901911] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 841.749508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 841.783529] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 842.821481] LNet: 40004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.834395] LNetError: 40004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 842.849270] LNet: Removed LNI 10.1.2.31@tcp [ 842.866640] LNet: Removed LNI 10.1.2.3@tcp [ 843.697586] Key type .llcrypt unregistered [ 843.699683] Key type ._llcrypt unregistered [ 857.105824] Key type ._llcrypt registered [ 857.108629] Key type .llcrypt registered [ 858.329214] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 00:11:07 (1790741467) [ 858.810981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 858.822346] alg: No test for adler32 (adler32-zlib) [ 859.707544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.787785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 859.836294] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 860.651816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.680262] LNet: 40955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.690906] LNetError: 40955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.704900] LNet: Removed LNI 192.168.202.28@tcp [ 860.736638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.810835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.28@tcp [ 860.836707] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 861.571652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.28@tcp [ 861.591839] LNet: Removed LNI 192.168.202.28@tcp [ 861.643598] LNet: 41090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.242916] Key type .llcrypt unregistered [ 862.246965] Key type ._llcrypt unregistered [ 874.706609] Key type ._llcrypt registered [ 874.714395] Key type .llcrypt registered [ 876.107760] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 00:11:25 (1790741485) [ 878.491035] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg228-server [ 880.418495] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 00:11:29 (1790741489) [ 880.878407] Key type .llcrypt unregistered [ 880.880472] Key type ._llcrypt unregistered [ 887.611948] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 887.836075] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 887.842311] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 887.847191] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 890.347305] Key type ._llcrypt registered [ 890.348962] Key type .llcrypt registered [ 890.504610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 890.531247] alg: No test for adler32 (adler32-zlib) [ 891.536406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.587710] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 892.187043] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 894.819164] LNet: 42926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.826850] LNetError: 42926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.830396] LNet: Removed LNI 192.168.202.28@tcp1 [ 894.852811] LNet: Removed LNI 10.1.2.3@tcp2 [ 895.437584] Key type .llcrypt unregistered [ 895.439926] Key type ._llcrypt unregistered [ 907.088504] Key type ._llcrypt registered [ 907.090952] Key type .llcrypt registered [ 908.179681] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 00:11:57 (1790741517) [ 908.677423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 908.705303] alg: No test for adler32 (adler32-zlib) [ 909.575722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.631131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 909.656806] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 910.328026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 910.348930] LNet: 43896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.359894] LNetError: 43896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.369474] LNet: Removed LNI 192.168.202.28@tcp [ 910.389322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.465218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 910.498323] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 916.470435] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 00:12:06 (1790741526) [ 916.836423] LNet: 44487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.839611] LNetError: 44487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.856730] LNet: Removed LNI 192.168.202.28@tcp [ 917.447545] Key type .llcrypt unregistered [ 917.450653] Key type ._llcrypt unregistered [ 924.121448] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 924.343176] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 924.347629] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 924.353831] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 926.839836] Key type ._llcrypt registered [ 926.841071] Key type .llcrypt registered [ 927.018526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 927.033427] alg: No test for adler32 (adler32-zlib) [ 928.107750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.235138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 928.282672] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 929.110875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 929.131781] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 931.394329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.431556] LNet: 45226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.437394] LNetError: 45226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 931.443910] LNet: Removed LNI 10.1.2.3@tcp [ 931.455041] LNet: Removed LNI 192.168.202.28@tcp [ 931.474519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.554774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 931.589917] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 931.597518] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 932.154177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.185837] LNet: 45253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.191878] LNetError: 45253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 932.196938] LNetError: 45253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 932.202834] LNet: Removed LNI 10.1.2.3@tcp [ 932.218791] LNet: Removed LNI 192.168.202.28@tcp [ 932.244636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.332474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 932.367633] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 932.372632] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 932.838183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 932.892538] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 934.218679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.236348] LNet: 45416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.249536] LNetError: 45416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 934.263924] LNetError: 45416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 934.274447] LNet: Removed LNI 10.1.2.3@tcp [ 934.276849] LNet: Removed LNI 192.168.202.28@tcp [ 934.287686] LNet: Removed LNI 192.168.202.28@tcp2 [ 934.316323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.420336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 934.475361] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 934.480267] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 934.489506] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 943.201390] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 00:12:32 (1790741552) [ 943.469090] LNet: 46010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.489264] LNetError: 46010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 943.503552] LNetError: 46010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 943.519139] LNet: Removed LNI 10.1.2.3@tcp [ 943.520622] LNet: Removed LNI 192.168.202.28@tcp [ 943.546988] LNet: Removed LNI 192.168.202.28@tcp2 [ 944.300758] Key type .llcrypt unregistered [ 944.305084] Key type ._llcrypt unregistered [ 951.329878] Key type ._llcrypt registered [ 951.331482] Key type .llcrypt registered [ 951.544169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 951.554586] alg: No test for adler32 (adler32-zlib) [ 952.526770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.612701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 952.660712] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 953.553373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.581621] LNet: 46406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.589802] LNetError: 46406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.619211] LNet: Removed LNI 192.168.202.28@tcp [ 953.654978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.715138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 953.732725] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 954.192436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.216235] LNet: 46436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.223390] LNetError: 46436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.235967] LNet: Removed LNI 192.168.202.28@tcp [ 954.256091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.302952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 954.326695] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 954.859995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.890976] LNet: Removed LNI 192.168.202.28@tcp [ 954.910756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.967561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 954.997632] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 955.598595] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 955.803286] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 955.808364] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 955.812051] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 958.324862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.357614] LNet: 46546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.366616] LNet: 46546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 958.374939] LNetError: 46546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.378093] LNetError: 46546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 958.401804] LNet: Removed LNI 192.168.202.28@tcp [ 958.429628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.498831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 958.520691] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 965.239634] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 00:12:54 (1790741574) [ 965.587336] LNet: 47145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.605237] LNetError: 47145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.640245] LNet: Removed LNI 192.168.202.28@tcp [ 966.207932] Key type .llcrypt unregistered [ 966.210450] Key type ._llcrypt unregistered [ 973.254719] Key type ._llcrypt registered [ 973.257228] Key type .llcrypt registered [ 973.490036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 973.520294] alg: No test for adler32 (adler32-zlib) [ 974.474387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.536515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 974.575217] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 975.267747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.283980] LNet: 47542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.293786] LNetError: 47542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.303791] LNet: Removed LNI 192.168.202.28@tcp [ 975.318858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.373362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 975.393195] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 982.739323] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 00:13:12 (1790741592) [ 983.062344] LNet: 48134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.070923] LNetError: 48134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.104902] LNet: Removed LNI 192.168.202.28@tcp [ 983.803699] Key type .llcrypt unregistered [ 983.805585] Key type ._llcrypt unregistered [ 991.340340] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 991.651191] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 991.655687] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 991.665735] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 994.413757] Key type ._llcrypt registered [ 994.415532] Key type .llcrypt registered [ 994.698628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 994.738283] alg: No test for adler32 (adler32-zlib) [ 995.760874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.870221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 995.898055] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 997.144161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.182725] LNet: 48580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.190044] LNetError: 48580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.222470] LNet: Removed LNI 192.168.202.28@tcp [ 997.246392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.312244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 997.351324] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1005.260596] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 00:13:34 (1790741614) [ 1005.488490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1005.525354] LNet: 49181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.531873] LNetError: 49181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.544233] LNet: Removed LNI 192.168.202.28@tcp [ 1005.563378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.683121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1011.746954] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 00:13:41 (1790741621) [ 1012.094685] LNet: 49763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.752518] Key type .llcrypt unregistered [ 1012.756534] Key type ._llcrypt unregistered [ 1020.102778] Key type ._llcrypt registered [ 1020.104574] Key type .llcrypt registered [ 1020.329454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1020.366091] alg: No test for adler32 (adler32-zlib) [ 1021.254750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1021.354520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1021.383844] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1022.278652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1022.300346] LNet: 50159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.307593] LNetError: 50159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.321385] LNet: Removed LNI 192.168.202.28@tcp [ 1022.339498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1022.396261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1022.423820] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1029.910591] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 00:13:58 (1790741638) [ 1030.036477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1030.072976] LNet: 50747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.079445] LNetError: 50747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.102128] LNet: Removed LNI 192.168.202.28@tcp [ 1030.161080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.211475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1036.601587] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 00:14:05 (1790741645) [ 1038.251528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1039.863979] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 00:14:09 (1790741649) [ 1041.855758] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1043.647980] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 00:14:12 (1790741652) [ 1045.254898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1046.845935] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 00:14:16 (1790741656) [ 1048.427828] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1050.395990] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 00:14:19 (1790741659) [ 1052.074838] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1053.985812] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 00:14:23 (1790741663) [ 1054.408421] LNet: 52778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.087883] Key type .llcrypt unregistered [ 1055.090594] Key type ._llcrypt unregistered [ 1062.294535] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1062.583171] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1062.593942] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1062.607453] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1065.203114] Key type ._llcrypt registered [ 1065.206198] Key type .llcrypt registered [ 1065.385840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1065.413499] alg: No test for adler32 (adler32-zlib) [ 1066.350792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1066.429493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1066.454344] LNet: Added LNI 192.168.202.28@tcp [8/512/0/180] [ 1066.460461] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1066.464356] LNet: 53097:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1066.566933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1066.590987] LNet: 53117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.604457] LNetError: 53117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1066.615487] LNet: Removed LNI 10.1.2.3@tcp [ 1066.616951] LNetError: 53117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.641458] LNet: Removed LNI 192.168.202.28@tcp [ 1066.673458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1066.737983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1066.760443] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1066.861847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1066.881401] LNet: 53141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.891331] LNetError: 53141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.917889] LNet: Removed LNI 192.168.202.28@tcp [ 1066.941773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1067.008741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1067.030331] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1067.034818] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1067.038330] LNet: 53149:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1067.117077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1067.137439] LNet: 53167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.142713] LNetError: 53167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1067.157764] LNet: Removed LNI 10.1.2.3@tcp [ 1067.162724] LNetError: 53167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.178452] LNet: Removed LNI 192.168.202.28@tcp [ 1067.194295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1067.253983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1067.269299] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1067.274409] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1067.276981] LNet: 53175:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1067.484289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1067.499934] LNet: 53201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.505275] LNetError: 53201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1067.510367] LNet: Removed LNI 10.1.2.3@tcp [ 1067.512657] LNetError: 53201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.523786] LNet: Removed LNI 192.168.202.28@tcp [ 1067.539660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1067.606411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1067.636194] LNet: Added LNI 192.168.202.28@tcp [8/256/0/182] [ 1067.639953] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1067.771425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1067.797226] LNet: 53229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.805221] LNetError: 53229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1067.817220] LNet: Removed LNI 10.1.2.3@tcp [ 1067.820695] LNetError: 53229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.843591] LNet: Removed LNI 192.168.202.28@tcp [ 1067.875513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1067.953422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1067.972362] LNet: Added LNI 192.168.202.28@tcp [8/256/0/182] [ 1067.975517] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1067.984376] LNet: 53237:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1068.092217] LNet: Removed LNI 10.1.2.3@tcp [ 1068.114781] LNet: Removed LNI 192.168.202.28@tcp [ 1068.741047] Key type .llcrypt unregistered [ 1068.743718] Key type ._llcrypt unregistered [ 1082.307396] Key type ._llcrypt registered [ 1082.310113] Key type .llcrypt registered [ 1083.833413] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 00:14:52 (1790741692) [ 1085.538637] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1087.634517] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 00:14:56 (1790741696) [ 1088.171945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1088.196647] alg: No test for adler32 (adler32-zlib) [ 1089.147678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1089.190660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1089.227661] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1090.175883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.28 [ 1090.212526] LNet: 54501:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1090.220511] LNet: Removed LNI 192.168.202.28@tcp [ 1097.568250] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 00:15:06 (1790741706) [ 1097.754856] LNet: 55064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.763836] LNetError: 55064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.795747] LNet: Removed LNI 192.168.202.28@tcp [ 1098.467575] Key type .llcrypt unregistered [ 1098.471237] Key type ._llcrypt unregistered [ 1105.163054] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1105.450246] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1108.158774] Key type ._llcrypt registered [ 1108.161893] Key type .llcrypt registered [ 1108.361481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1108.384367] alg: No test for adler32 (adler32-zlib) [ 1109.786741] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1116.638905] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 00:15:25 (1790741725) [ 1117.083369] LNet: 55983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.089766] LNetError: 55983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1117.102063] LNet: Removed LNI 10.1.2.3@tcp [ 1117.791770] Key type .llcrypt unregistered [ 1117.793560] Key type ._llcrypt unregistered [ 1123.933466] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1124.229951] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1124.233127] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1124.238966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1126.832613] Key type ._llcrypt registered [ 1126.834493] Key type .llcrypt registered [ 1127.053624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1127.071161] alg: No test for adler32 (adler32-zlib) [ 1128.073393] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1134.721886] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 00:15:43 (1790741743) [ 1136.172828] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1137.739321] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 00:15:47 (1790741747) [ 1137.988645] LNet: 57166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.007971] LNetError: 57166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1138.030412] LNet: Removed LNI 10.1.2.3@tcp [ 1138.703511] Key type .llcrypt unregistered [ 1138.705449] Key type ._llcrypt unregistered [ 1145.815215] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1146.131392] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1146.140803] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1146.146666] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1148.946682] Key type ._llcrypt registered [ 1148.948734] Key type .llcrypt registered [ 1149.209972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1149.237129] alg: No test for adler32 (adler32-zlib) [ 1150.216384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1150.292129] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1159.433282] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 00:16:08 (1790741768) [ 1160.265450] LNet: 58394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1160.279779] LNetError: 58394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1160.299899] LNet: Removed LNI 10.1.2.3@tcp [ 1160.945791] Key type .llcrypt unregistered [ 1160.950151] Key type ._llcrypt unregistered [ 1166.420113] Key type ._llcrypt registered [ 1166.423674] Key type .llcrypt registered [ 1166.631759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1166.649887] alg: No test for adler32 (adler32-zlib) [ 1167.586429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1167.630234] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1175.687513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1176.494793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_interrupt -r 1 [ 1176.532916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_interrupt -r 1 [ 1176.586583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1176.614180] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1176.864207] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1176.903795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.943095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1176.981926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.023613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.056990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1177.099409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.136386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.197593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.258517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.295512] LNet: 58833:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1177.325961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.364665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.398731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.429706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.470555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.504929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.534422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1177.996664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_dropped -r 1 [ 1178.020432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_dropped -r 1 [ 1178.040786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1178.054270] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1178.216220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1178.254352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1178.278906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1178.305587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.329397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1178.345513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.372737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.403708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.425596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.450838] LNet: 58944:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1178.462899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1178.480809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.498887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.520178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1178.539644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1178.566961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.595190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1179.162820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_aborted -r 1 [ 1179.217450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_aborted -r 1 [ 1179.249539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1179.265397] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1179.478484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.495280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1179.517941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.542361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.559510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1179.577993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.590588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1179.607453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1179.641629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1179.666355] LNet: 59055:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1179.694658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1179.724223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1179.762098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1179.791476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.845532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.881505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1179.924953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1180.354452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_no_route -r 1 [ 1180.398976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_no_route -r 1 [ 1180.445989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1180.643555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.667620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1180.689938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1180.713173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1180.734891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1180.766895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1180.796736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.822397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.845532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.860064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.885820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.923646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.970931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.019063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.064777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.092710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1181.574962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_timeout -r 1 [ 1181.607190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_timeout -r 1 [ 1181.656500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1181.680659] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1181.683797] LNet: Skipped 1 previous similar message [ 1181.937348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.958580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.977587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.000083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.019752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1182.035807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.061408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.082394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.107485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.123466] LNet: 59278:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.132427] LNet: 59278:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1182.149919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.179754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.207737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.238686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.268873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.298932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.322352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1182.689289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_error -r 1 [ 1182.718721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_error -r 1 [ 1182.755363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1182.988711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.027721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.070455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.109470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.148405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1183.171876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.196787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.233419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.268775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.292162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.320475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.365474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.402582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.441579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.480136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.518177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1183.745320] LNet: 59430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.753098] LNetError: 59430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.807804] LNet: Removed LNI 192.168.202.28@tcp [ 1185.159556] Key type .llcrypt unregistered [ 1185.166136] Key type ._llcrypt unregistered [ 1207.302836] Key type ._llcrypt registered [ 1207.303988] Key type .llcrypt registered [ 1208.519586] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 00:16:57 (1790741817) [ 1209.689405] Key type .llcrypt unregistered [ 1209.694587] Key type ._llcrypt unregistered [ 1218.174342] Key type ._llcrypt registered [ 1218.183405] Key type .llcrypt registered [ 1218.445864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1218.462602] alg: No test for adler32 (adler32-zlib) [ 1219.640769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1219.744334] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1228.033432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1238.253335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1238.312187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1238.341256] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1239.338903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1240.157852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_interrupt -r 1 [ 1240.198336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_interrupt -r 1 [ 1240.240669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_interrupt -r 1 [ 1240.280078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_interrupt -r 1 [ 1240.330552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_interrupt -r 1 [ 1240.380803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_interrupt -r 1 [ 1240.428970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_interrupt -r 1 [ 1240.483593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_interrupt -r 1 [ 1240.534558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1240.568193] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1240.578850] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp1 [ 1240.961613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.993683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1241.025794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.052752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.084991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1241.131834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.166963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.195183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.225632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.244908] LNet: 60904:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.266409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.295990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1241.330389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1241.360924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1241.406233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.454224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1241.486970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1242.165881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_dropped -r 1 [ 1242.194431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_dropped -r 1 [ 1242.229679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_dropped -r 1 [ 1242.261922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_dropped -r 1 [ 1242.294482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_dropped -r 1 [ 1242.325710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_dropped -r 1 [ 1242.363489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_dropped -r 1 [ 1242.402246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_dropped -r 1 [ 1242.448451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1242.470091] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1242.473402] LNet: Skipped 2 previous similar messages [ 1242.757621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1242.811555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1242.835616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1242.868496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1242.905597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1242.945737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.002476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.047494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.095683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.113230] LNet: 61028:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.136905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.161856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.186821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.213885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.236924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.270753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.302588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1244.038352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_aborted -r 1 [ 1244.075783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_aborted -r 1 [ 1244.117809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_aborted -r 1 [ 1244.168711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_aborted -r 1 [ 1244.211425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_aborted -r 1 [ 1244.269936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_aborted -r 1 [ 1244.318720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_aborted -r 1 [ 1244.370596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_aborted -r 1 [ 1244.424284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1244.450428] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1244.457778] LNet: Skipped 2 previous similar messages [ 1244.781517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1244.813164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1244.841092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1244.872561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1244.910747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1244.979052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1245.037615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1245.068941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.102397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1245.120174] LNet: 61151:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.134374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1245.162188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.192139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.217879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.242959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.287919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1245.333870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1245.921715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_no_route -r 1 [ 1245.969918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_no_route -r 1 [ 1246.023153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_no_route -r 1 [ 1246.096328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_no_route -r 1 [ 1246.167320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_no_route -r 1 [ 1246.239619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_no_route -r 1 [ 1246.316571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_no_route -r 1 [ 1246.376586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_no_route -r 1 [ 1246.423425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1246.450856] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1246.454108] LNet: Skipped 2 previous similar messages [ 1246.822558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1246.860758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1246.915831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1246.960340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1247.011395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1247.070424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1247.113645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.150274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.191254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.213884] LNet: 61274:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.253914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.305794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.338056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.376400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.409644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1247.442920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1247.489978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1248.117391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_timeout -r 1 [ 1248.154223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_timeout -r 1 [ 1248.209550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_timeout -r 1 [ 1248.259593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_timeout -r 1 [ 1248.315422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_timeout -r 1 [ 1248.387785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_timeout -r 1 [ 1248.440695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_timeout -r 1 [ 1248.509708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_timeout -r 1 [ 1248.565162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1248.881659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1248.934546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1248.963652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1248.989798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.020712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1249.049977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.089732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.151471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.199989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.245915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.287281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.329565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.387364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.466541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.493420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1249.528786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1250.116336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_error -r 1 [ 1250.142700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_error -r 1 [ 1250.170426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_error -r 1 [ 1250.196732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_error -r 1 [ 1250.221765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_error -r 1 [ 1250.247982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_error -r 1 [ 1250.278549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_error -r 1 [ 1250.304397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_error -r 1 [ 1250.339804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1250.655884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1250.709549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1250.744807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1250.777986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1250.809184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1250.850475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1250.896441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1250.951616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1250.992649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1251.037336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1251.086653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1251.123287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1251.158516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1251.197314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1251.231059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1251.272761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1256.305413] LNet: 61596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1256.313375] LNetError: 61596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1256.321678] LNet: Removed LNI 192.168.202.28@tcp [ 1256.331613] LNet: Removed LNI 192.168.202.28@tcp1 [ 1256.869587] Key type .llcrypt unregistered [ 1256.872497] Key type ._llcrypt unregistered [ 1274.501948] Key type ._llcrypt registered [ 1274.503439] Key type .llcrypt registered [ 1275.979126] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 00:18:04 (1790741884) [ 1277.005725] Key type .llcrypt unregistered [ 1277.007451] Key type ._llcrypt unregistered [ 1283.697749] Key type ._llcrypt registered [ 1283.699811] Key type .llcrypt registered [ 1284.085378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1284.115406] alg: No test for adler32 (adler32-zlib) [ 1285.104226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1285.125863] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1292.729593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1293.667393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_dropped -r 1 [ 1293.729300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_dropped -r 1 [ 1293.781880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1293.806867] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1294.144661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1294.206994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1294.241900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1294.310533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1294.356066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1294.391550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1294.418966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1294.450072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1294.482352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1294.511655] LNet: 62846:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1294.531894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1294.562981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1294.604581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1294.645127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1294.683925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1294.744848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1294.792412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1295.337492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_error -r 1 [ 1295.370771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_error -r 1 [ 1295.436637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1295.466661] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1295.715996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1295.749952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1295.778284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1295.802160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1295.826975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1295.851557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1295.876447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1295.901899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1295.930956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1295.948208] LNet: 62958:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1295.962412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1295.985813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1296.010942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1296.035502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1296.059623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.082312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1296.108641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1296.576167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_timeout -r 1 [ 1296.603610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_timeout -r 1 [ 1296.644981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1296.662283] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1296.860644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1296.888916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1296.929406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1296.952192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1296.984962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1297.019961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1297.049565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1297.072340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1297.095196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1297.108338] LNet: 63069:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1297.125275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1297.147808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1297.175285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1297.198189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1297.224571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.250818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1297.274467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1297.522659] LNet: 63110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.529298] LNetError: 63110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1297.545536] LNet: Removed LNI 192.168.202.28@tcp [ 1297.934650] Key type .llcrypt unregistered [ 1297.938281] Key type ._llcrypt unregistered [ 1317.913621] Key type ._llcrypt registered [ 1317.915657] Key type .llcrypt registered [ 1319.503935] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 00:18:48 (1790741928) [ 1320.667402] Key type .llcrypt unregistered [ 1320.669049] Key type ._llcrypt unregistered [ 1330.563074] Key type ._llcrypt registered [ 1330.565702] Key type .llcrypt registered [ 1331.089484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1331.138286] alg: No test for adler32 (adler32-zlib) [ 1332.286691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1332.338469] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1342.529348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1353.045370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1353.094307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1353.121471] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1354.121471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1355.071925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_dropped -r 1 [ 1355.135551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e remote_dropped -r 1 [ 1355.239914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_dropped -r 1 [ 1355.346818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e remote_dropped -r 1 [ 1355.448498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e remote_dropped -r 1 [ 1355.538699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e remote_dropped -r 1 [ 1355.613931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e remote_dropped -r 1 [ 1355.678561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e remote_dropped -r 1 [ 1355.719925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1355.757909] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1355.768159] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp1 [ 1356.028446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1356.050820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1356.082203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1356.113801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1356.150898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1356.178565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1356.204911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1356.236747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1356.267601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1356.289964] LNet: 64584:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1356.309398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1356.345373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1356.387250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1356.429477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1356.455879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1356.487937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1356.513854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1356.541333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1357.008355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_error -r 1 [ 1357.040528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e remote_error -r 1 [ 1357.069488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_error -r 1 [ 1357.108354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e remote_error -r 1 [ 1357.156450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e remote_error -r 1 [ 1357.202532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e remote_error -r 1 [ 1357.243607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e remote_error -r 1 [ 1357.287587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e remote_error -r 1 [ 1357.325104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1357.347464] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1357.354297] LNet: Skipped 2 previous similar messages [ 1357.675466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1357.710688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1357.738441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1357.770614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1357.791788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1357.826247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1357.850481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1357.885446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1357.923902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1357.940569] LNet: 64709:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1357.962400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1357.991090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1358.024361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1358.061537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1358.088828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1358.120652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1358.152341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1358.178444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1358.685646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_timeout -r 1 [ 1358.711413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e remote_timeout -r 1 [ 1358.744718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_timeout -r 1 [ 1358.773121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e remote_timeout -r 1 [ 1358.813267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e remote_timeout -r 1 [ 1358.853544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e remote_timeout -r 1 [ 1358.905456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e remote_timeout -r 1 [ 1358.958735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e remote_timeout -r 1 [ 1358.997681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1359.019355] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1359.274270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1359.329829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1359.359121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1359.380757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1359.436216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1359.475496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1359.508390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1359.566718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1359.626146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1359.649862] LNet: 64835:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1359.676817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1359.742738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1359.777108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1359.817695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1359.851675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1359.893686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1359.933054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1359.975521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1365.698743] LNet: 64913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.705843] LNetError: 64913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1366.751404] LNet: Removed LNI 192.168.202.28@tcp [ 1366.753743] LNetError: 64913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1366.782330] LNet: Removed LNI 192.168.202.28@tcp1 [ 1367.496971] Key type .llcrypt unregistered [ 1367.501830] Key type ._llcrypt unregistered [ 1386.031612] Key type ._llcrypt registered [ 1386.032981] Key type .llcrypt registered [ 1387.217390] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 00:19:56 (1790741996) [ 1389.298202] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1391.180948] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 00:20:00 (1790742000) [ 1392.525768] Key type .llcrypt unregistered [ 1392.530556] Key type ._llcrypt unregistered [ 1399.352459] Key type ._llcrypt registered [ 1399.356573] Key type .llcrypt registered [ 1399.599581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1399.625871] alg: No test for adler32 (adler32-zlib) [ 1400.568160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1400.603436] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1408.901605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1409.742587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e network_timeout -r 1 [ 1409.792325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e network_timeout -r 1 [ 1409.872486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1409.906436] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1410.135599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1410.156676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1410.185568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1410.226215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1410.246897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1410.263892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1410.286641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1410.301972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1410.316668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1410.324544] LNet: 66459:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1410.332256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1410.346975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1410.363608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1410.378277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1410.403457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1410.419729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1410.535551] LNet: 66498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.538558] LNetError: 66498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.546479] LNet: Removed LNI 192.168.202.28@tcp [ 1410.807477] Key type .llcrypt unregistered [ 1410.809354] Key type ._llcrypt unregistered [ 1432.014718] Key type ._llcrypt registered [ 1432.017944] Key type .llcrypt registered [ 1432.217300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1432.229284] alg: No test for adler32 (adler32-zlib) [ 1433.285698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1433.324080] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1443.813969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1452.284716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1452.352473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1452.387444] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1453.700478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1454.617495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e network_timeout -r 1 [ 1454.661811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e network_timeout -r 1 [ 1454.743441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e network_timeout -r 1 [ 1454.808337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e network_timeout -r 1 [ 1454.892507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e network_timeout -r 1 [ 1454.930671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e network_timeout -r 1 [ 1454.980490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e network_timeout -r 1 [ 1455.020489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e network_timeout -r 1 [ 1455.061428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1455.088443] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1455.410773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1455.503788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1455.589934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1455.655471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1455.732967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1455.789278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1455.841642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1455.876811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1455.903128] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp1 [ 1455.905950] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp1 [ 1455.924744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1455.951207] LNet: 67382:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1455.990996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1456.070898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1456.153479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1456.185844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1456.236662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1456.320790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1461.555960] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1461.937983] LNet: 67457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.945678] LNetError: 67457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.953487] LNet: Removed LNI 192.168.202.28@tcp [ 1461.969435] LNet: Removed LNI 192.168.202.28@tcp1 [ 1462.353800] Key type .llcrypt unregistered [ 1462.360656] Key type ._llcrypt unregistered [ 1479.307799] Key type ._llcrypt registered [ 1479.309803] Key type .llcrypt registered [ 1480.518459] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 00:21:29 (1790742089) [ 1480.939268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1480.966397] alg: No test for adler32 (adler32-zlib) [ 1481.891112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.946595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1481.977783] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1482.701676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1482.733613] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1483.510595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1483.553724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1483.590429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1483.618696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1483.646496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1483.671286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1483.697497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.28@tcp [ 1483.711704] LNetError: 68555:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.202.28@tcp to recovery [ 1483.999582] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1483.999620] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1485.023590] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1486.222292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1489.119472] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1490.007080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1490.035572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1490.084467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1490.112667] LNet: 68651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1490.121063] LNetError: 68651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1490.127306] LNet: Removed LNI 192.168.202.28@tcp [ 1490.144701] LNet: Removed LNI 192.168.202.28@tcp1 [ 1490.169756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.240424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1490.274392] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1490.869347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1490.885563] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1491.650166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1491.681509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1491.728226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1491.765649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1491.804300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1491.852296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1491.900181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1491.945993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.28@tcp [ 1491.981378] LNetError: 68924:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.202.28@tcp to recovery [ 1492.257855] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1495.580390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1498.402105] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1498.413680] LNet: Skipped 1 previous similar message [ 1503.685452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1503.710365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1503.729710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1511.281738] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 00:22:00 (1790742120) [ 1511.615544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.635272] LNet: 69640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.646299] LNetError: 69640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.651221] LNetError: 69640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1511.656254] LNet: Removed LNI 192.168.202.28@tcp [ 1511.668491] LNet: Removed LNI 192.168.202.28@tcp1 [ 1511.711529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.788119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1511.807615] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1512.555675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1512.574659] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1513.302288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1513.389344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1513.450423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1513.488690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1513.524515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1513.554209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1513.583343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.28@tcp --health 0 [ 1513.677641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1513.825082] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.28@tcp [ 1514.848524] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1514.855454] LNet: Skipped 1 previous similar message [ 1529.169950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1529.337542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1529.365970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1529.400770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.28@tcp --health 500 [ 1529.546214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1536.342561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1536.427908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1536.459534] LNet: 70132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.466498] LNetError: 70132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1536.474324] LNetError: 70132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1536.484791] LNet: Removed LNI 192.168.202.28@tcp [ 1536.505304] LNet: Removed LNI 192.168.202.28@tcp1 [ 1536.524156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.578021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1536.605352] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1537.417696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1537.436418] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1538.344344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1538.385915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1538.454823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1538.490703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1538.526754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1538.557654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1538.594427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1538.624575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.28@tcp --health 0 [ 1539.616721] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1539.628111] LNet: Skipped 3 previous similar messages [ 1540.162890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1550.993992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1551.079319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1551.162980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1559.789778] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 00:22:48 (1790742168) [ 1560.732781] LNet: 71180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.741185] LNetError: 71180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1560.745677] LNetError: 71180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1561.759421] LNet: Removed LNI 192.168.202.28@tcp [ 1561.782190] LNet: Removed LNI 192.168.202.28@tcp1 [ 1562.585733] Key type .llcrypt unregistered [ 1562.594787] Key type ._llcrypt unregistered [ 1569.863804] Key type ._llcrypt registered [ 1569.866404] Key type .llcrypt registered [ 1570.231714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1570.278625] alg: No test for adler32 (adler32-zlib) [ 1571.291278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1571.310698] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1581.801761] Lustre: DEBUG MARKER: Initial discovery [ 1581.847049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.128@tcp [ 1583.880363] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1584.388393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1584.440862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.128@tcp [ 1584.453357] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1584.466826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1584.497497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.128@tcp -r 1 -m GET -l 3 [ 1584.554591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.128@tcp -r 1 -m GET -e local_error [ 1584.592253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.128@tcp -r 1 -m PUT -l 6 [ 1584.608682] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1586.181182] Lustre: DEBUG MARKER: Force oleg228-client.virtnet to discover 192.168.202.128@tcp (in background) [ 1586.468135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.128@tcp [ 1587.777582] Lustre: DEBUG MARKER: Force oleg228-server to discover 192.168.202.28@tcp [ 1589.536843] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1590.898962] Lustre: DEBUG MARKER: Wait for 71945 [ 1592.579154] Lustre: DEBUG MARKER: Finished wait on 71945 [ 1592.676150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1592.741785] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1592.747508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1593.026302] LNet: 72393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.041059] LNetError: 72393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.112705] LNet: Removed LNI 192.168.202.28@tcp [ 1594.724521] Key type .llcrypt unregistered [ 1594.726980] Key type ._llcrypt unregistered [ 1612.042697] Key type ._llcrypt registered [ 1612.045417] Key type .llcrypt registered [ 1613.475670] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 00:23:42 (1790742222) [ 1614.181444] Key type .llcrypt unregistered [ 1614.183499] Key type ._llcrypt unregistered [ 1622.022681] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1622.180152] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1622.191157] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1622.205630] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1624.990944] Key type ._llcrypt registered [ 1624.994793] Key type .llcrypt registered [ 1625.278252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1625.303991] alg: No test for adler32 (adler32-zlib) [ 1626.325767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.401440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1626.436240] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1627.289557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1627.307866] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1635.689185] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 00:24:05 (1790742245) [ 1635.871488] LNet: 74472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1635.890065] LNetError: 74472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1635.903981] LNet: Removed LNI 10.1.2.3@tcp [ 1635.925483] LNet: Removed LNI 192.168.202.28@tcp [ 1636.614850] Key type .llcrypt unregistered [ 1636.617982] Key type ._llcrypt unregistered [ 1642.893698] Key type ._llcrypt registered [ 1642.896406] Key type .llcrypt registered [ 1643.137657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1643.160632] alg: No test for adler32 (adler32-zlib) [ 1644.160988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.272622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1644.317276] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1645.157188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1645.179743] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1646.209443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.28@tcp [ 1647.264180] LNetError: Unexpected error -22 connecting to 192.168.202.28@tcp at host 192.168.202.28:988 [ 1652.958590] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 00:24:22 (1790742262) [ 1653.061347] LNet: 75594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.064731] LNetError: 75594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1654.111338] LNet: Removed LNI 10.1.2.3@tcp [ 1654.115471] LNetError: 75594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.137382] LNet: Removed LNI 192.168.202.28@tcp [ 1654.703784] Key type .llcrypt unregistered [ 1654.705468] Key type ._llcrypt unregistered [ 1660.293885] Key type ._llcrypt registered [ 1660.295461] Key type .llcrypt registered [ 1660.577478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1660.599636] alg: No test for adler32 (adler32-zlib) [ 1661.556546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.638528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1661.688172] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1662.380498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1662.402807] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 1663.299923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.28@tcp1 --nid 192.168.202.28@tcp2 [ 1667.420524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp1 192.168.202.28@tcp2 [ 1667.433349] LNetError: 76436:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.28@tcp1 to 192.168.202.28@tcp2 - Route table may be misconfigured [ 1667.447623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp2 192.168.202.28@tcp1 [ 1673.359318] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 00:24:42 (1790742282) [ 1673.489957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1673.538214] LNet: 77001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1673.573057] LNetError: 77001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.594183] LNet: Removed LNI 192.168.202.28@tcp1 [ 1674.604152] LNetError: 77001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.629857] LNet: Removed LNI 192.168.202.28@tcp2 [ 1674.667226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.734345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1674.762759] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1675.493752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1675.510572] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1676.329462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1676.356761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.28@tcp [ 1676.384373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp1 192.168.202.28@tcp1 [ 1676.424788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.28@tcp -d 192.168.202.28@tcp -e network_timeout [ 1676.459993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -e network_timeout [ 1676.493265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -e network_timeout [ 1676.520828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -e network_timeout [ 1676.540836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1676.557351] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1676.586914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1676.695292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1676.768343] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1676.787306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1682.978967] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 00:24:52 (1790742292) [ 1683.271226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.316200] LNet: 77870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.334175] LNetError: 77870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.383170] LNet: Removed LNI 192.168.202.28@tcp [ 1685.416226] LNet: Removed LNI 192.168.202.28@tcp1 [ 1685.449423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.543211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1685.850228] LNet: 77907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.295911] Key type .llcrypt unregistered [ 1686.298069] Key type ._llcrypt unregistered [ 1697.916292] Key type ._llcrypt registered [ 1697.918023] Key type .llcrypt registered [ 1699.122084] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 00:25:08 (1790742308) [ 1699.395756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1699.414446] alg: No test for adler32 (adler32-zlib) [ 1700.320367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.531737] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1703.494041] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 00:25:12 (1790742312) [ 1703.584596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.610323] LNet: 79000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.638387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.709761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.726611] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1704.262362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1704.276677] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1705.003328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1705.070499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp1 [ 1705.128129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp1 [ 1711.709326] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 00:25:20 (1790742320) [ 1713.543198] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg228-server [ 1715.087751] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 00:25:24 (1790742324) [ 1716.429517] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg228-server [ 1717.940879] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 00:25:27 (1790742327) [ 1719.590904] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg228-server [ 1721.477970] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 00:25:30 (1790742330) [ 1724.124531] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg228-server [ 1726.491299] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 00:25:35 (1790742335) [ 1729.045237] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg228-server [ 1731.383923] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 00:25:40 (1790742340) [ 1733.848862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg228-server [ 1735.968332] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 00:25:44 (1790742344) [ 1738.874539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg228-server [ 1741.161942] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 00:25:49 (1790742349) [ 1744.034362] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg228-server [ 1745.639870] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 00:25:55 (1790742355) [ 1748.116528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg228-server [ 1749.784698] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 00:25:59 (1790742359) [ 1749.927795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.966060] LNet: 82880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.976601] LNetError: 82880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.994628] LNet: Removed LNI 192.168.202.28@tcp [ 1751.023254] LNet: Removed LNI 192.168.202.28@tcp1 [ 1751.134886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.264919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1751.312690] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1752.647303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1752.729865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1753.040900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.065590] LNet: 83034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.071892] LNetError: 83034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.076367] LNetError: 83034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1754.157469] LNet: Removed LNI 192.168.202.28@tcp [ 1754.233279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.332312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1754.369213] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1755.218487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1755.488743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.533198] LNet: 83179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.551770] LNetError: 83179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.576733] LNet: Removed LNI 192.168.202.28@tcp [ 1755.602368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.669944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1755.685819] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1756.252701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1756.373116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1756.706731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.798821] LNet: Removed LNI 192.168.202.28@tcp [ 1757.876953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.060196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1758.128300] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1759.361761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1760.742029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.783038] LNet: 83481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.790379] LNet: 83481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1760.816759] LNetError: 83481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.822626] LNetError: 83481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1760.841294] LNet: Removed LNI 192.168.202.28@tcp [ 1760.886858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.980217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.005222] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1762.143601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1762.238996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1762.476567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.557783] LNet: Removed LNI 192.168.202.28@tcp [ 1763.583914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.684218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1763.749737] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1765.112143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1765.394933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.436609] LNetError: 83773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.443478] LNetError: 83773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1766.501153] LNet: Removed LNI 192.168.202.28@tcp [ 1766.526393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.605846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1766.650740] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1767.389490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1767.520980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1767.841638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.934616] LNet: Removed LNI 192.168.202.28@tcp [ 1768.987292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.056474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1769.123220] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1770.230724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1770.617377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.646134] LNet: 84066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.653725] LNet: 84066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1771.686947] LNet: Removed LNI 192.168.202.28@tcp [ 1771.728593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.848902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1771.890325] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1772.941614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1773.019644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1773.285277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.377746] LNet: Removed LNI 192.168.202.28@tcp [ 1774.410732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.474167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1774.526353] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1775.683335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1776.115686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.154648] LNetError: 84359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.170586] LNetError: 84359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1777.268947] LNet: Removed LNI 192.168.202.28@tcp [ 1777.297129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.429516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1777.492256] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1778.190811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1778.300281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1778.583537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.685436] LNet: Removed LNI 192.168.202.28@tcp [ 1779.723482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.817540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1779.874082] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1781.114044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1781.591551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.649806] LNet: Removed LNI 192.168.202.28@tcp [ 1782.698160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.847616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1782.901504] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1783.759933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1783.834485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.890147] LNet: Removed LNI 192.168.202.28@tcp [ 1783.943641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.054150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1784.123254] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1784.675597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1784.779635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.816209] LNet: Removed LNI 192.168.202.28@tcp [ 1784.844897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.948114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1785.084432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.151472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.279165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1785.411179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.454798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.512902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1785.636121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.686050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.752686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1785.782974] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1786.747394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.772331] LNet: 85091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.778769] LNet: 85091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1786.798355] LNet: Removed LNI 192.168.202.28@tcp [ 1786.826186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.914866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1796.770712] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 00:26:45 (1790742405) [ 1797.262328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1797.320786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.418555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1797.458895] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 1799.658635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.682799] LNetError: 85811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.685368] LNetError: 85811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1799.707806] LNet: Removed LNI 192.168.202.28@tcp231 [ 1799.726424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.808699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1799.857950] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 1799.862643] LNet: 85819:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1800.553033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1800.599661] LNet: Removed LNI 192.168.202.28@tcp231 [ 1800.629286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1800.675580] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 1801.964167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1802.036661] LNet: Removed LNI 192.168.202.28@tcp231 [ 1802.073574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.150883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1802.183803] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/0] [ 1802.190858] LNet: 85972:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1802.562778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1802.593204] LNet: Removed LNI 192.168.202.28@tcp231 [ 1802.632042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1802.681018] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/0] [ 1804.182653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.223628] LNet: Removed LNI 192.168.202.28@tcp231 [ 1804.246889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.308579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1804.339328] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/60] [ 1804.349655] LNet: 86123:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1804.879300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1804.906237] LNet: Removed LNI 192.168.202.28@tcp231 [ 1804.920451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1804.952428] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/60] [ 1806.390915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.448721] LNet: Removed LNI 192.168.202.28@tcp231 [ 1806.471386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.572821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1806.608632] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 1806.615158] LNet: 86274:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1807.020079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1807.049633] LNet: Removed LNI 192.168.202.28@tcp231 [ 1807.075987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1807.129593] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 1816.245623] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 00:27:05 (1790742425) [ 1816.472715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1816.548712] LNet: Removed LNI 192.168.202.28@tcp231 [ 1816.617451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.684703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1816.710867] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 1817.537878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1817.622487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1817.669089] LNet: Removed LNI 192.168.202.28@tcp232 [ 1817.691234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1817.730616] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 1819.048642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1819.126301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1819.185463] LNet: Removed LNI 192.168.202.28@tcp232 [ 1819.202994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1819.269977] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 1820.658819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1820.753583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1820.829748] LNet: Removed LNI 192.168.202.28@tcp232 [ 1820.865183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1820.929753] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 1822.416880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1822.490113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1822.542691] LNet: Removed LNI 192.168.202.28@tcp232 [ 1822.581958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1831.446365] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 00:27:20 (1790742440) [ 1832.112229] LNet: 88113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.122831] LNet: 88113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1833.026881] Key type .llcrypt unregistered [ 1833.029469] Key type ._llcrypt unregistered [ 1839.309523] Key type ._llcrypt registered [ 1839.312571] Key type .llcrypt registered [ 1839.547726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1839.564262] alg: No test for adler32 (adler32-zlib) [ 1840.477541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1840.537145] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1848.276160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1859.950832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1860.032876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1860.089382] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1861.114910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1862.266425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_interrupt -r 2 [ 1862.347442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_interrupt -r 2 [ 1862.460160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_interrupt -r 2 [ 1862.539138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_interrupt -r 2 [ 1862.615971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_interrupt -r 2 [ 1862.727756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_interrupt -r 2 [ 1862.819385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_interrupt -r 2 [ 1862.928642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_interrupt -r 2 [ 1863.020338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1863.077269] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1863.091136] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1863.511634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1863.534348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1863.564408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1863.658579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1863.706930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1863.734932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1863.766666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1863.805654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1863.860019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1863.880411] LNet: 88780:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1863.927757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1863.980242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1864.024512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1864.062818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1864.127970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1864.177972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1864.784723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_dropped -r 2 [ 1864.806721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_dropped -r 2 [ 1864.838654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_dropped -r 2 [ 1864.867295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_dropped -r 2 [ 1864.895694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_dropped -r 2 [ 1864.920895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_dropped -r 2 [ 1864.956871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_dropped -r 2 [ 1864.985180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_dropped -r 2 [ 1865.014416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1865.031149] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1865.307237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1865.356286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1865.392812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1865.427900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1865.477938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1865.524676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1865.591175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1865.659750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1865.736174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1865.778774] LNet: 88903:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1865.833731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1865.864343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1865.891956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1865.919973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1865.958717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1865.990712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1866.730997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_aborted -r 2 [ 1866.754428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_aborted -r 2 [ 1866.802935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_aborted -r 2 [ 1866.831068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_aborted -r 2 [ 1866.879665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_aborted -r 2 [ 1866.921394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_aborted -r 2 [ 1866.987579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_aborted -r 2 [ 1867.030683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_aborted -r 2 [ 1867.075595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1867.096192] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1867.479719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1867.530665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1867.573960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1867.648702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1867.685082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1867.720674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1867.759249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1867.803945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1867.893683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1867.922518] LNet: 89025:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1867.980572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1868.048250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1868.096452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1868.151345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1868.196495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1868.258735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1868.847333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_no_route -r 2 [ 1868.890559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_no_route -r 2 [ 1868.931375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_no_route -r 2 [ 1868.983928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_no_route -r 2 [ 1869.067607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_no_route -r 2 [ 1869.135636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_no_route -r 2 [ 1869.245730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_no_route -r 2 [ 1869.300591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_no_route -r 2 [ 1869.334293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1869.374726] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1869.639333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1869.679647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1869.705975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1869.738701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1869.778983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1869.819519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1869.852345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1869.880527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1869.899298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1869.914504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1869.972515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1870.022106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1870.073983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1870.144249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1870.190884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1870.932732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_timeout -r 2 [ 1870.956411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_timeout -r 2 [ 1870.992301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_timeout -r 2 [ 1871.022516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_timeout -r 2 [ 1871.068686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_timeout -r 2 [ 1871.117936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_timeout -r 2 [ 1871.185486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_timeout -r 2 [ 1871.224230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_timeout -r 2 [ 1871.285960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1871.630965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1871.653451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1871.686323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1871.737809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1871.772246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1871.802464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1871.841062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1871.886337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1871.917391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1871.931538] LNet: 89270:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1871.936356] LNet: 89270:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1871.953791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1871.978276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1872.002757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1872.034347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1872.085611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1872.131613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1872.694333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_dropped -r 2 [ 1872.745919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e remote_dropped -r 2 [ 1872.781664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_dropped -r 2 [ 1872.822385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e remote_dropped -r 2 [ 1872.849882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e remote_dropped -r 2 [ 1872.892852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e remote_dropped -r 2 [ 1872.918987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e remote_dropped -r 2 [ 1872.946937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e remote_dropped -r 2 [ 1872.973831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1872.997147] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp [ 1873.197213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1873.241812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1873.301309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1873.342656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1873.411558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1873.475624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1873.524544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1873.598181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1873.669588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1873.716954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1873.791481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1873.829694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1873.885471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1873.947368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1873.995035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1880.220681] LNet: 89466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.227305] LNetError: 89466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.234748] LNet: Removed LNI 192.168.202.28@tcp [ 1880.258465] LNet: Removed LNI 192.168.202.28@tcp1 [ 1880.766808] Key type .llcrypt unregistered [ 1880.770217] Key type ._llcrypt unregistered [ 1898.810883] Key type ._llcrypt registered [ 1898.812916] Key type .llcrypt registered [ 1900.194857] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 00:28:29 (1790742509) [ 1900.618054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1900.640835] alg: No test for adler32 (adler32-zlib) [ 1901.619657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.677364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1901.696820] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1902.816086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1902.874606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.28@tcp [ 1902.902843] LNetError: 90444:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.202.28@tcp to recovery [ 1903.019042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.28@tcp [ 1903.156743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1910.516034] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 00:28:39 (1790742519) [ 1910.818491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1910.856439] LNet: 91033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.868323] LNetError: 91033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.883833] LNet: Removed LNI 192.168.202.28@tcp [ 1910.908854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.963978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1910.985386] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1911.705495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1911.723511] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1912.626995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.28@tcp [ 1912.644272] LNetError: 91300:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.202.28@tcp to recovery [ 1912.991635] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1912.994715] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1913.877669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.28@tcp [ 1921.972989] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 00:28:50 (1790742530) [ 1923.895410] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg228-server [ 1925.976544] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 00:28:54 (1790742534) [ 1928.165635] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg228-server [ 1929.801571] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 00:28:59 (1790742539) [ 1931.661740] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg228-server [ 1933.382358] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 00:29:02 (1790742542) [ 1933.516891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1933.567866] LNet: 92917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.577202] LNetError: 92917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1933.587728] LNet: Removed LNI 192.168.202.28@tcp [ 1933.603498] LNet: Removed LNI 192.168.202.28@tcp1 [ 1933.644517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1933.707844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1933.725716] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1934.444410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1934.509231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1934.541655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1934.646727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1934.686960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1934.749652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1940.176966] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 00:29:09 (1790742549) [ 1940.418856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1940.444526] LNet: 93654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1940.451978] LNetError: 93654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1940.463635] LNetError: 93654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1940.481535] LNet: Removed LNI 192.168.202.28@tcp [ 1940.498736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1940.548310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1940.578077] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1947.399288] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 00:29:16 (1790742556) [ 1947.735245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1947.762683] LNet: 94357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1947.768779] LNetError: 94357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1947.780330] LNet: Removed LNI 192.168.202.28@tcp [ 1947.813990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.917711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1947.943771] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1955.997198] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 00:29:24 (1790742564) [ 1957.536723] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1958.986810] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 00:29:28 (1790742568) [ 1959.703839] LNet: 95392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1959.714486] LNetError: 95392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1959.735313] LNet: Removed LNI 192.168.202.28@tcp [ 1960.398447] Key type .llcrypt unregistered [ 1960.401940] Key type ._llcrypt unregistered [ 1968.982073] Key type ._llcrypt registered [ 1968.984115] Key type .llcrypt registered [ 1969.301229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1969.332939] alg: No test for adler32 (adler32-zlib) [ 1970.311428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1970.340900] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1978.198683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1983.546913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.128@tcp [ 1984.079222] LNet: 95809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1984.090703] LNetError: 95809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1984.119557] LNet: Removed LNI 192.168.202.28@tcp [ 1984.658544] Key type .llcrypt unregistered [ 1984.660492] Key type ._llcrypt unregistered [ 1996.574658] Key type ._llcrypt registered [ 1996.578032] Key type .llcrypt registered [ 1998.255455] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1999.702426] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2001.371423] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 00:30:10 (1790742610) [ 2003.270820] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2005.394463] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 00:30:14 (1790742614) [ 2007.481497] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg228-server [ 2009.723951] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 00:30:18 (1790742618) [ 2012.028960] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg228-server [ 2014.286817] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 00:30:23 (1790742623) [ 2016.016556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2016.053479] alg: No test for adler32 (adler32-zlib) [ 2017.121651] Lustre: Lustre: Build Version: 2.17.59_3_g21e240e [ 2017.365856] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2019.192771] Key type lgssc registered [ 2020.886708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2041.380440] Key type lgssc unregistered [ 2041.668804] LNet: 98992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.675825] LNetError: 98992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.700167] LNet: Removed LNI 192.168.202.28@tcp [ 2042.632656] Key type .llcrypt unregistered [ 2042.634460] Key type ._llcrypt unregistered [ 2057.039544] Key type ._llcrypt registered [ 2057.044586] Key type .llcrypt registered [ 2058.499508] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 00:31:07 (1790742667) [ 2058.924391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2058.952736] alg: No test for adler32 (adler32-zlib) [ 2059.920488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.987221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2060.047917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2060.103725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2060.153834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2067.123261] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 00:31:16 (1790742676) [ 2067.200787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2067.242891] LNet: 100408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2067.270216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2067.358783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2067.380232] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2068.208809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2068.228689] LNet: 100557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.232737] LNetError: 100557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.241367] LNet: Removed LNI 192.168.202.28@tcp [ 2068.254348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2068.322947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2068.372942] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2069.115781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2069.134442] LNetError: 100586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2069.148333] LNet: Removed LNI 192.168.202.28@tcp [ 2069.168368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2069.228225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2069.280033] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2077.216982] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 00:31:26 (1790742686) [ 2077.357151] LNet: 101167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2077.360522] LNet: 101167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2077.376487] LNetError: 101167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2077.410689] LNet: Removed LNI 192.168.202.28@tcp [ 2078.327827] Key type .llcrypt unregistered [ 2078.334534] Key type ._llcrypt unregistered [ 2085.723775] Key type ._llcrypt registered [ 2085.726309] Key type .llcrypt registered [ 2086.096784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2086.147313] alg: No test for adler32 (adler32-zlib) [ 2087.205222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2087.339844] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2087.344962] LNet: *** cfs_fail_loc=e003, val=100*** [ 2088.326739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2088.343254] LNet: *** cfs_fail_loc=e003, val=100*** [ 2088.345932] LNet: Skipped 2 previous similar messages [ 2088.351156] Ping reply received with mapping: [ 2088.352753] found metadata reply NI[192.168.202.28@tcp] [ 2088.357282] 1c ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2088.362774] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2088.370689] 5a 5a 5a 5a 5a 5a 5a 5a [ 2088.539095] LNet: 101573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.545503] LNetError: 101573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2088.587824] LNet: Removed LNI 192.168.202.28@tcp [ 2089.233509] Key type .llcrypt unregistered [ 2089.236727] Key type ._llcrypt unregistered [ 2103.605776] Key type ._llcrypt registered [ 2103.614890] Key type .llcrypt registered [ 2105.549521] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 00:31:53 (1790742713) [ 2106.424358] Key type .llcrypt unregistered [ 2106.434528] Key type ._llcrypt unregistered [ 2115.334780] Key type ._llcrypt registered [ 2115.336721] Key type .llcrypt registered [ 2115.623354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2115.666819] alg: No test for adler32 (adler32-zlib) [ 2116.910334] LNetError: 102614:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2117.420272] Key type .llcrypt unregistered [ 2117.425833] Key type ._llcrypt unregistered [ 2124.814289] Key type ._llcrypt registered [ 2124.816770] Key type .llcrypt registered [ 2126.399693] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 00:32:15 (1790742735) [ 2127.001497] Key type .llcrypt unregistered [ 2127.004284] Key type ._llcrypt unregistered [ 2133.977101] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2133.982497] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2136.810808] Key type ._llcrypt registered [ 2136.813065] Key type .llcrypt registered [ 2137.149289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2137.181307] alg: No test for adler32 (adler32-zlib) [ 2138.194275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2138.259986] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2150.086673] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 00:32:39 (1790742759) [ 2150.233613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2150.266616] LNet: 104386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.277656] LNetError: 104386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2150.329694] LNet: Removed LNI 10.1.2.3@tcp [ 2150.386741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.750613] Key type .llcrypt unregistered [ 2151.752453] Key type ._llcrypt unregistered [ 2160.046572] Key type ._llcrypt registered [ 2160.052636] Key type .llcrypt registered [ 2160.495138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2160.534691] alg: No test for adler32 (adler32-zlib) [ 2161.484901] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2161.622298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2161.798329] LNet: 104680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.809339] LNetError: 104680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2162.881624] LNet: Removed LNI 192.168.202.28@tcp [ 2163.934406] Key type .llcrypt unregistered [ 2163.940054] Key type ._llcrypt unregistered [ 2179.359319] Key type ._llcrypt registered [ 2179.364208] Key type .llcrypt registered [ 2181.398227] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 00:33:09 (1790742789) [ 2182.110489] Key type .llcrypt unregistered [ 2182.112370] Key type ._llcrypt unregistered [ 2189.698359] Key type ._llcrypt registered [ 2189.701533] Key type .llcrypt registered [ 2190.104335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2190.132656] alg: No test for adler32 (adler32-zlib) [ 2197.541774] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 00:33:26 (1790742806) [ 2197.787494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2197.852542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2197.988335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2198.010030] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2198.762330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2198.797597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2198.858765] LNet: Removed LNI 192.168.202.28@tcp [ 2199.103422] LNet: 106423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2199.628469] Key type .llcrypt unregistered [ 2199.636408] Key type ._llcrypt unregistered [ 2212.657978] Key type ._llcrypt registered [ 2212.660799] Key type .llcrypt registered [ 2214.306264] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 00:33:43 (1790742823) [ 2214.812821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2214.832788] alg: No test for adler32 (adler32-zlib) [ 2215.836489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2215.891151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2215.909728] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2216.845279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2216.881380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.28@tcp [ 2216.900839] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2216.905477] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2216.908768] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2216, 0, 0 [ 2216.913544] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2216.916351] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2216.919986] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2216.922786] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2216, 0, 0 [ 2216.928844] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2216.931058] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2216.935021] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2216.938484] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2216, 0, 2266 [ 2216.942581] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2216.944880] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2216.948662] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2216.951512] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2216, 0, 2266 [ 2216.954621] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2224.084297] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 00:33:53 (1790742833) [ 2224.556657] LNet: 107962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2224.566175] LNetError: 107962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2224.582719] LNet: Removed LNI 192.168.202.28@tcp [ 2225.230513] Key type .llcrypt unregistered [ 2225.232550] Key type ._llcrypt unregistered [ 2231.108968] Key type ._llcrypt registered [ 2231.114561] Key type .llcrypt registered [ 2231.356136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2231.369599] alg: No test for adler32 (adler32-zlib) [ 2232.311599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2232.335810] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2239.747516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 2248.391542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.449774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2248.480297] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 2249.360705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 2249.636429] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2249.960099] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2249.982308] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2249.992705] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2252.474774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2252.528112] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2264.000238] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 00:34:33 (1790742873) [ 2264.225377] LNet: 109472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2264.259480] LNetError: 109472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2264.269401] LNet: Removed LNI 192.168.202.28@tcp [ 2265.312542] LNet: Removed LNI 192.168.202.28@tcp1 [ 2265.325260] LNetError: 109472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2265.335343] LNetError: 109472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2265.359297] LNet: Removed LNI 10.1.2.3@tcp99 [ 2266.296666] Key type .llcrypt unregistered [ 2266.299472] Key type ._llcrypt unregistered [ 2273.237379] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2273.242837] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2275.936342] Key type ._llcrypt registered [ 2275.939907] Key type .llcrypt registered [ 2276.188835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2276.249766] alg: No test for adler32 (adler32-zlib) [ 2277.171380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2277.221236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2277.251179] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2278.164399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2278.191206] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2281.335743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2281.437761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2281.526303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2281.567737] LNetError: 110216:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2281.620298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2281.685792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2282.044341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2282.134951] LNet: 110236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2282.142876] LNetError: 110236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2282.150122] LNet: Removed LNI 10.1.2.3@tcp [ 2283.181233] LNet: Removed LNI 192.168.202.28@tcp [ 2283.976484] Key type .llcrypt unregistered [ 2283.980466] Key type ._llcrypt unregistered [ 2297.479096] Key type ._llcrypt registered [ 2297.481333] Key type .llcrypt registered [ 2299.196763] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 00:35:07 (1790742907) [ 2299.598502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2299.607665] alg: No test for adler32 (adler32-zlib) [ 2300.467854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2300.531869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2300.547628] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2301.029752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2301.108177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg228-client@tcp [ 2308.555964] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 00:35:17 (1790742917) [ 2308.690949] LNet: 111762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2308.713741] LNetError: 111762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2308.746117] LNet: Removed LNI 192.168.202.28@tcp [ 2309.540267] Key type .llcrypt unregistered [ 2309.541909] Key type ._llcrypt unregistered [ 2317.344528] Key type ._llcrypt registered [ 2317.346696] Key type .llcrypt registered [ 2317.595465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2317.621713] alg: No test for adler32 (adler32-zlib) [ 2318.617968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2318.667123] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2318.724138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2326.480562] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 00:35:35 (1790742935) [ 2328.115930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2330.331341] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 00:35:39 (1790742939) [ 2331.894819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2333.891564] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 00:35:42 (1790742942) [ 2334.057863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.098643] LNet: 113185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2334.104807] LNetError: 113185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2334.123460] LNet: Removed LNI 192.168.202.28@tcp [ 2334.185808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.333380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2334.378924] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2335.457481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2335.587885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2342.608285] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 00:35:51 (1790742951) [ 2342.756470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2342.809306] LNet: 113902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2342.827490] LNetError: 113902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2342.856472] LNet: Removed LNI 192.168.202.28@tcp [ 2342.897116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2342.946849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2342.975385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2343.577779] Key type .llcrypt unregistered [ 2343.580169] Key type ._llcrypt unregistered [ 2354.888226] Key type ._llcrypt registered [ 2354.891734] Key type .llcrypt registered [ 2356.365643] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 00:36:05 (1790742965) [ 2356.804548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2356.829849] alg: No test for adler32 (adler32-zlib) [ 2357.764150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2357.840155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2357.870685] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2358.573076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2358.644335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2364.972756] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 00:36:14 (1790742974) [ 2365.064970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.085515] LNet: 115451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2365.089534] LNetError: 115451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2365.107557] LNet: Removed LNI 192.168.202.28@tcp [ 2365.141913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.217848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2365.257628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2372.009965] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 00:36:21 (1790742981) [ 2372.253673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2372.282824] LNet: 116037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2372.336525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2372.479509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2372.970511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2373.015292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2373.105167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2373.647805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2373.681438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2373.741153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2373.756812] LNet: 116070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2373.778791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2373.844857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2374.253803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2374.289250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2374.360858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2374.894338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2374.931623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2382.028331] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 00:36:30 (1790742990) [ 2382.117906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2382.143471] LNet: 116663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2382.200245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2382.264680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2382.310305] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2382.878896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2382.895453] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2383.563887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2383.628424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2383.838838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2383.863435] LNetError: 116950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2384.934746] LNet: Removed LNI 192.168.202.28@tcp [ 2384.936651] LNetError: 116950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2384.970740] LNet: Removed LNI 192.168.202.28@tcp2 [ 2385.037652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2385.127126] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2385.140964] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2385.146688] LNet: 116956:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2385.668963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2385.963801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2387.039406] LNet: Removed LNI 192.168.202.28@tcp [ 2387.043384] LNetError: 116993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2387.060874] LNetError: 116993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2387.083450] LNet: Removed LNI 192.168.202.28@tcp2 [ 2387.114538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2387.191563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2387.211719] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2388.024404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2388.051247] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2389.014526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2389.045310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2389.098337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2389.323775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2389.341227] LNet: 117278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2389.345682] LNet: 117278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2390.370084] LNet: Removed LNI 192.168.202.28@tcp [ 2391.404066] LNet: Removed LNI 192.168.202.28@tcp2 [ 2391.457063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2391.563713] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2391.572840] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2391.577852] LNet: 117284:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2392.210088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2392.408306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.430467] LNetError: 117321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2392.438446] LNetError: 117321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2392.443739] LNet: Removed LNI 192.168.202.28@tcp [ 2393.525815] LNet: Removed LNI 192.168.202.28@tcp2 [ 2393.553766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2393.602468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2393.631706] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2394.433344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2394.461566] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2395.344961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2395.466444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2395.780491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2396.831173] LNet: Removed LNI 192.168.202.28@tcp [ 2396.853966] LNet: Removed LNI 192.168.202.28@tcp2 [ 2396.890052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2396.977992] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2396.996944] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2397.005348] LNet: 117607:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2397.742625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2397.976374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2397.992520] LNet: 117644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2397.999942] LNet: 117644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2399.073521] LNet: Removed LNI 192.168.202.28@tcp [ 2399.095076] LNet: Removed LNI 192.168.202.28@tcp2 [ 2399.124929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2399.203735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2399.235407] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2399.978676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2399.994801] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2400.928380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2400.973263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2401.047935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2401.298654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2401.317984] LNetError: 117928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2401.323299] LNetError: 117928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2401.328510] LNet: Removed LNI 192.168.202.28@tcp [ 2402.362667] LNet: Removed LNI 192.168.202.28@tcp2 [ 2402.421684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2402.506916] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2402.516865] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2402.523717] LNet: 117934:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2403.192270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2403.530352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2403.581231] LNet: Removed LNI 192.168.202.28@tcp [ 2404.650654] LNet: Removed LNI 192.168.202.28@tcp2 [ 2404.723329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2404.848964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2404.916961] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2405.900388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2405.934037] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2406.821340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2406.854605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2406.913201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2407.392716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2408.479186] LNet: Removed LNI 192.168.202.28@tcp [ 2408.515943] LNet: Removed LNI 192.168.202.28@tcp2 [ 2408.535546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2408.594935] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2408.599422] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2408.603608] LNet: 118274:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2409.271162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2409.575597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2410.655298] LNet: Removed LNI 192.168.202.28@tcp [ 2410.674981] LNet: Removed LNI 192.168.202.28@tcp2 [ 2410.725451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2410.806980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2410.843606] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2411.698967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2411.741640] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2412.740846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2412.798414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2412.890940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2412.967326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2413.411853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2414.433172] LNet: Removed LNI 192.168.202.28@tcp [ 2415.465228] LNet: Removed LNI 192.168.202.28@tcp2 [ 2415.524762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2415.607357] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2415.611141] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2415.979291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2421.988766] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2423.398463] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 00:37:12 (1790743032) [ 2423.479891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2423.501452] LNet: 119387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2423.506522] LNet: 119387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2423.514839] LNetError: 119387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2423.523899] LNetError: 119387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2423.529853] LNet: Removed LNI 192.168.202.28@tcp [ 2424.551277] LNet: Removed LNI 192.168.202.28@tcp2 [ 2424.596305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2424.947189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2424.970839] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2425.680918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2425.719073] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2426.592788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2426.612652] LNet: Removed LNI 10.1.2.3@tcp [ 2426.623444] LNet: Removed LNI 192.168.202.28@tcp [ 2426.644171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2426.706429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2426.732507] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2426.737045] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2427.125809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2427.142830] LNet: Removed LNI 192.168.202.28@tcp [ 2427.154586] LNet: Removed LNI 10.1.2.3@tcp [ 2432.265890] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 00:37:21 (1790743041) [ 2432.399651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2432.451782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2432.505092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2432.537977] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2433.110445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2433.170242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2433.225740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2433.256423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2433.291424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2433.332585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2433.369919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2433.406547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2433.442443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2433.475688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2433.502136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2433.529388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2433.675489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2439.325893] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 00:37:28 (1790743048) [ 2439.388985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2440.488902] LNet: Removed LNI 192.168.202.28@tcp [ 2440.527477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.631216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2440.671965] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2441.462561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2441.508064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2441.610372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2441.678063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2441.746779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2441.794748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2441.850109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2441.917817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2441.965159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2442.004898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2442.048740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2442.111963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2442.233742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2442.274603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2442.308710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2442.352954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2442.394359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2442.428585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2442.459785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2442.501084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2442.537588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2442.572461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2442.606865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2449.191431] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 00:37:38 (1790743058) [ 2449.302462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2450.345518] LNet: Removed LNI 192.168.202.28@tcp [ 2450.395768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2450.512881] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2450.673780] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2450.676580] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2450.680081] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2453.001509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2453.025003] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2453.968256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2453.987838] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2455.781797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2455.810575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2455.874398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.28@tcp -m PUT -e local_timeout -r 1 [ 2455.948850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2456.007682] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2456.011824] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2456.250424] LNetError: 121828:0:(lib-move.c:4794:lnet_parse()) 192.168.202.28@tcp, src 192.168.202.28@tcp: Dropping PUT (error -108 looking up sender) [ 2457.312406] LNet: 122286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2457.317620] LNet: 122286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2457.332309] LNetError: 122286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2457.340388] LNetError: 122286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2457.349631] LNet: Removed LNI 10.1.2.3@tcp [ 2458.408816] LNet: Removed LNI 192.168.202.28@tcp [ 2458.785203] Key type .llcrypt unregistered [ 2458.788073] Key type ._llcrypt unregistered [ 2469.480231] Key type ._llcrypt registered [ 2469.488886] Key type .llcrypt registered [ 2470.905029] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 00:37:59 (1790743079) [ 2471.250749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2471.266919] alg: No test for adler32 (adler32-zlib) [ 2472.172588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2472.506746] LNet: 123137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2473.162586] Key type .llcrypt unregistered [ 2473.168770] Key type ._llcrypt unregistered [ 2479.587578] Key type ._llcrypt registered [ 2479.593294] Key type .llcrypt registered [ 2479.817321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2479.837233] alg: No test for adler32 (adler32-zlib) [ 2480.801906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2480.828784] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2488.502435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 2488.689202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2489.261463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.128@tcp [ 2489.543904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2489.578880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2489.612180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2489.642509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2489.673488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2489.707408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2489.735848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2489.770365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2489.825862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2489.847472] LNet: 123586:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2489.872191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2489.914486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2489.951254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2489.980494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2490.032192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2490.066325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2490.743276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2490.976185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2490.993065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2491.014754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2491.034555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2491.054962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2491.077788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2491.104219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2491.105829] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp [ 2491.141205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2491.171775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2491.191478] LNet: 123706:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2491.215589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2491.251559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2491.278569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2491.321813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2491.354911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2491.382436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2491.938302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.128@tcp [ 2492.284449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2492.311463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2492.340300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2492.378211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2492.413542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2492.463861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2492.497288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2492.533284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2492.581144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2492.603476] LNet: 123825:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2492.644088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2492.679433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2492.741661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2492.781681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2492.808873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2492.845977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2493.582984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2493.859230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2493.890989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2493.918499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2493.958561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2494.021363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2494.068533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2494.118155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2494.168456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2494.211616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2494.255938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2494.298522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2494.341208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2494.399932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2494.445432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2494.486423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2494.867336] LNet: 123990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2494.872468] LNetError: 123990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2495.912278] LNet: Removed LNI 192.168.202.28@tcp [ 2496.705738] Key type .llcrypt unregistered [ 2496.708419] Key type ._llcrypt unregistered [ 2513.865930] Key type ._llcrypt registered [ 2513.871426] Key type .llcrypt registered [ 2514.958070] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 00:38:44 (1790743124) [ 2515.406954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2515.416238] alg: No test for adler32 (adler32-zlib) [ 2516.319506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2516.735334] LNet: 124876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2517.294261] Key type .llcrypt unregistered [ 2517.297550] Key type ._llcrypt unregistered [ 2523.992164] Key type ._llcrypt registered [ 2523.994566] Key type .llcrypt registered [ 2524.191506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2524.211241] alg: No test for adler32 (adler32-zlib) [ 2525.175071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2525.198894] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2531.796499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 2541.298547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2541.344767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2541.372406] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 2542.394323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 2542.518433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2543.082400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.128@tcp [ 2543.321923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2543.341370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2543.370343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2543.402927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2543.453944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2543.506220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2543.541755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2543.564888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2543.592360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2543.616484] LNet: 125535:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2543.641727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2543.649829] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp [ 2543.700710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2543.749633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2543.797913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2543.842432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2543.875849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2544.531322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2544.737494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2544.762844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2544.789223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2544.812403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2544.841614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2544.879391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2544.926521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2545.000176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2545.048188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2545.077057] LNet: 125654:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2545.119423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2545.152348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2545.182315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2545.218783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2545.261760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2545.308878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2546.009556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.128@tcp [ 2546.233845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2546.280275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2546.329102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2546.361801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2546.399630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2546.427733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2546.456977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2546.488169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2546.522633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2546.551757] LNet: 125773:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2546.584898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2546.662588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2546.711769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2546.750386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2546.813490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2546.883891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2547.813144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2548.029830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2548.070899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2548.124557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2548.153807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2548.178804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2548.211808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2548.237130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2548.263648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2548.292527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2548.316572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2548.361611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2548.407690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2548.439771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2548.472612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2548.511297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2553.264713] LNet: 125973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2553.269920] LNetError: 125973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2554.335434] LNet: Removed LNI 192.168.202.28@tcp [ 2554.340326] LNetError: 125973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2554.374521] LNet: Removed LNI 192.168.202.28@tcp1 [ 2554.780728] Key type .llcrypt unregistered [ 2554.782688] Key type ._llcrypt unregistered [ 2570.620227] Key type ._llcrypt registered [ 2570.621517] Key type .llcrypt registered [ 2571.848259] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 00:39:41 (1790743181) [ 2572.265836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2572.290310] alg: No test for adler32 (adler32-zlib) [ 2573.298720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2573.362949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2573.402110] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2574.185587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2574.205793] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2575.014730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2575.109178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2576.159451] LNet: Removed LNI 192.168.202.28@tcp [ 2582.652914] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 00:39:51 (1790743191) [ 2584.210186] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2585.957334] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 00:39:55 (1790743195) [ 2592.752916] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 00:40:02 (1790743202) [ 2599.934797] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 00:40:09 (1790743209) [ 2606.374793] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 00:40:15 (1790743215) [ 2613.908986] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 00:40:23 (1790743223) [ 2622.528448] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 00:40:31 (1790743231) [ 2630.609481] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 00:40:39 (1790743239) [ 2638.090269] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 00:40:47 (1790743247) [ 2644.992856] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 00:40:54 (1790743254) [ 2651.521892] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 00:41:00 (1790743260) [ 2657.810619] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 00:41:07 (1790743267) [ 2664.715246] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 00:41:14 (1790743274) [ 2671.024881] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 00:41:20 (1790743280) [ 2676.869461] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 00:41:26 (1790743286) [ 2684.037230] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 00:41:33 (1790743293) [ 2691.731254] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 00:41:40 (1790743300) [ 2698.912191] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 00:41:48 (1790743308) [ 2705.705744] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 00:41:55 (1790743315) [ 2705.918377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2705.938406] LNet: 137662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2705.947267] LNetError: 137662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2706.002724] LNet: Removed LNI 192.168.202.28@tcp2 [ 2706.029759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2706.169090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2706.189572] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2707.346220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2707.371225] LNet: 137806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2707.378994] LNetError: 137806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2707.405779] LNet: Removed LNI 192.168.202.28@tcp [ 2707.431506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2707.499372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2707.517122] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2713.874612] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 00:42:03 (1790743323) [ 2720.717954] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 00:42:09 (1790743329) [ 2728.195177] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 00:42:17 (1790743337) [ 2737.126569] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 00:42:26 (1790743346) [ 2745.026481] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 00:42:34 (1790743354) [ 2753.136905] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 00:42:42 (1790743362) [ 2761.251383] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 00:42:50 (1790743370) [ 2768.241718] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 00:42:57 (1790743377) [ 2775.274460] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 00:43:04 (1790743384) [ 2782.629723] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 00:43:11 (1790743391) [ 2790.306169] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 00:43:19 (1790743399) [ 2799.031540] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 00:43:27 (1790743407) [ 2806.618746] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2529 sec ========== 00:43:35 (1790743415) [ 2807.033255] LNet: 145272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2807.037269] LNetError: 145272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2807.051052] LNet: Removed LNI 192.168.202.28@tcp [ 2807.854711] Key type .llcrypt unregistered [ 2807.856149] Key type ._llcrypt unregistered [ 2821.652505] Key type ._llcrypt registered [ 2821.654135] Key type .llcrypt registered [ 2822.292705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2822.310435] alg: No test for adler32 (adler32-zlib) [ 2823.679735] Lustre: Lustre: Build Version: 2.17.59_3_g21e240e [ 2823.950911] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2825.647670] Key type lgssc registered [ 2827.475238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2887.626777] Lustre: Mounted lustre-client - version 2.17.59_3_g21e240e [ 2893.122555] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2900.374946] Lustre: Unmounted lustre-client [ 2919.472838] Key type lgssc unregistered [ 2919.780257] LNet: 148227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2919.790754] LNetError: 148227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2920.807162] LNet: Removed LNI 192.168.202.28@tcp [ 2921.839241] Key type .llcrypt unregistered [ 2921.843105] Key type ._llcrypt unregistered